There are a few things to remember when designing custom layouts for HO scale model railroad trains. First - enjoy yourself! There is lots of room for creativity, just stay with in a few basic guidelines.
Some very useful online tools including free software to design your own railroad model layout; search for it - it is even fun to use. You can also do it the "old fashioned" way, using pencil and paper with a compass for curves and a ruler for straight-aways. Keep in mind that no matter how exact you are in your design, when it comes to laying the materials, some adjustments will always be necessary.
You can also design your layout right on location, placing track pieces and switches loosely in place to get a feel for what it will look like. You can move things around until you have it exactly the way you want it. This is known as "going free-lance."
It is helpful to deliberately leave openings for later changes and additions. Leave some space for a train yard you might be able to add later, or other details you might not have the time or money for in your railroad model right now. This makes for a rewarding, multi-stage project, growing in complexity over time.
There are several different types of yards you can add. A "hump" yard has a gentle slope with an uncoupler, allowing uncoupled cars to gently roll away from the mainline. A staging yard is a section of track deliberately hidden from view, where you send your trains to wait. It is useful to simulate long-distance runs, or simply for storage purposes.
One important guideline to keep in mind is the minimum radius for your turns, which in HO scale model trains is 18 inches for 4 axle diesels and 22 inches for 6 axle diesels. As a general rule, things will look and work best with 24 inch radius turns. For a full circle, this means 48 inches diameter, so you will need slightly more than a 4 foot wide space for such a layout.
Another important guideline is the maximum grade, which for HO scale model trains is 4%. What this means is that you should take 100 horizontal inches to rise 4 vertical inches. And that's a maximum. Try to keep hills somewhat less steep than that. And test your engines to find out what they can handle and how many cars they can pull up that grade you're creating.
To simulate greater height, instead of creating a steep climb for the rails themselves, drop the ground level underneath. You can create some breathtaking gorge and valley scenery, or even just send a road or another track underneath.
For getting started with your first layout, it's probably best to stick with sectional track, which is fairly easy to snap together. More advanced modelers of HO scale model trains prefer to construct their layout from flex track. It comes in bundles, so it's generally cheaper, and, as its name implies, it is a very versatile option, with greater opportunity for customization. It will make your railroad model look good. You can create small, brief curves with more natural lead-ins as opposed to being restricted to standard radii and straightaways in the snap-together sets.
A final point to consider in your design is to be sure you will be able to get to all parts of the finished layout for maintenance purposes. Some advanced modelers constructing larger layouts will even make "access hatches", large holes covered by a movable portion of scenery, where they can come up from beneath and actually stand in the middle to work on those less accessible portions. Be sure the opening is large enough to comfortably stand and move around in. The scenery on the removable section should be light and durable for ease of handling. You may also want to create a screen of trees or some other visual obstruction in front of the removable section.
