CASE STUDY
High-Complexity Foliage
Part 1
Dissecting the Real-Time Foliage Problem
Creating believable foliage is notoriously difficult, but forcing dense vegetation to respect tight real-time performance budgets is where the real engineering begins.
While real-time rendering now powers the pinnacle of visual content delivery, foliage remains one of the hardest assets to scale effectively.
Part 1 of this study breaks down the end-to-end foliage pipeline through three key angles:
The Realism Gap: What gives foliage its natural complexity, and why traditional real-time rendering struggles to replicate it.
The Production Pipeline: The step-by-step process from initial reference to final asset.
Engine & Shader Setup: x
Contents of this study:
01 — Real vs. Real-Time
What makes foliage so tricky
02 — The Asset Pipeline
Breaking down the required elements
03 — Final Assembly
Importing/Assembling the final asset in Unreal Engine
04 — Shader Setup
Understanding the important role of a good shader
05 — Part 2
Note that this is not a tutorial on creating foliage, but rather a complete, surface-level look at the methods and processes behind creating real-time foliage.
Rendered in Unreal Engine 5
Real vs. Real-Time
One of the reasons why foliage, especially trees are so tricky to render correctly is due to their volumetric nature.
Unlike static objects like rocks, the engine has to accurately simulate light piercing through layers and layers of information, while ensuring that each time it passes through something it calculates the next pass-through correctly. If this is done improperly, we lose depth.
There are primarily three rendering features that drive depth:
Shadowing
Ambient occlusion
Subsurface scattering
01
Light Interaction
Shadows
Ambient occlusion
Subsurface scattering
Subsurface scattering simulates how light passes through an object. This is core to almost any foliage because of the thin and organic nature of it.
Subsurface self-shadowing is an interesting feature to play with as it controls how much lighting pierces the individual layers of an object.
As you can see in the image below, a lower value means more light will pass through the tree. It can cause some depth to be lost, so be careful when adjusting this.
Shape & Structure
Why real-time engines struggle
The Asset Pipeline
3D Scanning
These parts of the tree are generally captured using photogrammetry to achieve a clean, delit and highly detailed asset.
Trunks
Its important to capture a wide range of shapes and sizes when creating a complete species, as it has to encompass the species from sapling to adult to broken/fallen.
Include both easily re-usable shapes and more unique ones to break up the repetition. Try to also look for some interesting color & pattern variations.
Besides being used to build the tree, from an environment creation POV many of these elements are the foundation of a forest, and will be used as ground scatter.
That is also why we make sure to capture closed meshes, as we at times use simulation to create a volumetric and layered forest base.
Tileable surfaces are derived from the trunk scans. The standard aspect ratio for bark is 2:1, but sometimes 4:1 is used on thin branches.
Elements
Surfaces
Textures
Use Mosaic
Meshing
Test case: Pinus halepensis
Processing
RAW images
With a proper cross-polarized camera setup and good, consistent shot-taking, there is generally little that needs to be done to the RAW images. However, because it is always better to do adjustments in RAW, some common tweaks include:
Exposure & Highlights
Vignette — due to the nature of using a ring flash combined with depth, the edges of the image tend to be darker. We can easily counteract this, and it will help generate a flat color/albedo.
Vignette removal, before/after.
Software
Metashape has over the years been our primary photogrammetry engine.
One of the realities of capturing in the wild is that you will probably not get a perfect dataset, and that is why reliability is the most important factor. Metashape handles data variance exceptionally well, which helps reduce the number of bad scans to near 0.
Other features such as rolling shutter compensation for drones with a rolling shutter can be a literal scan-saver when capturing in rough conditions.
However, Metashape is not a one-click problem solver. It requires more time to process and is more limited in terms of automation, which can lead to more time being spent cleaning up the scans.
Natural texture blending over sharpens and can create stretching and other errors. It also takes significantly longer, and baking 16K is extremely resource intensive.
02
Don’t enable hole filling
Closing small holes
Alignment
Align at a lower quality
You will use it to help know which areas are missing textures.
Small holes are often hard to detect. We first close the holes, and then selectively smooth the source mesh.
When alignment fails you are in trouble. Despite mitigating this through good on-location practices, nothing is perfect.
There are a few tricks we can employ to try and save it.
Optimizations don't really work
This may be a hot take, but we never found optimizations helpful from the POV of fixing serious alignment issues, and we tried a lot of them.
We’re not saying it can’t help fix minor issues, but if your scan has serious issues, there is a good chance you won’t be able to fix it (though we still try and sometimes succeed!)
You generally shouldn’t be using highest quality for aligning because it upscales the images. It can in fact produce worse results.
If you are having minor alignment issues, or some of your images won’t align, lowering the quality to medium or even low can help, though we keep it on high unless there is a serious issue.
Lower depth maps quality
Object masking
If you are seeing alignment errors on your mesh, try processing an extra mesh on a lower depth map quality.
You can then use that mesh as your lowpoly base, bake and patch up any visible texture errors, or accept a less detailed source (or bake both and blend textures).
This method creates a mask for your source images of the exact subject you are scanning, and may help alignment. We have saved a few scans using this technique, so it’s worth trying.
It involves a few potential approaches, either:
Build a mesh on a lower quality and using that to automatically create the masks in MS
Import your own masks
Automate this in Photoshop by creating an action applying focus mask and running the action in the image processor
Generate image masks using AI
Works well for things with a solid background, ie. turntable scanning. Far too unreliable to use for outdoor scans
Atlasing
Photometric stereo
This is the current gold standard for capturing foliage. It involves using lights in a controlled environment to capture the foliage being lit from different angles.
Once set up, it’s rather quick to get results but it does require additional equipment.
Some of the equipment—such as a proper LED backlight—can be tricky to get hold of and travel with, but capturing subsurface scattering is a nice bonus, but not a requirement.
Arrange your leaves
Get off your computer and play a IRL game of UV packing!
These examples don’t show it well, but if you also pack to fit your aspect ratio you can avoid repacking later.
Keep it clean
Do your best to remove dirt and other unwanted things from your atlas space. It will save you time down the line.
Black, light absorbing cloth
When capturing translucent surfaces, having a light and or reflective surface as a background will cause additional light bouncing when its being hit by a strong light. This can have adverse effects, so using a black light absorbing cloth tries to get around this problem.
The main challenge for foliage in this scenario would be when wanting to also capture SSS, as switching from cloth to panel is tricky.
Musou black has some cool products we’ve used quite a few times, albeit mainly for scanning smaller objects on a turntable. It makes the auto-masking process in Metashape quick and makes it easy to do a fully closed scan.
Still some issues
SSS reflections
When capturing SSS using backlight, areas where the edges are heavily bent down towards the light will be lit up disproportionally.
Can be fixed manually, but annoying nevertheless.
The equipment
To do color/normal/SSS capture, you need quite a bit of additional equipment. If your only goal is to capture 2D atlases, then its more manageable, but if you also go with cameras, flashes, drones and other accessories it becomes a lot.
One thing we have learned over the years is doing environment scan trips for weeks at a time can be tough.
Making sure that we travel without as light as possible becomes an important logistical consideration.
When capturing the normal map, if the leaf has a very vertically undulating shape you risk causing a translucency effect on areas catching the light from the side.
The solution is usually to move your light source up and avoid strong lighting perpendicular to the leaf, but in some cases it can be hard to avoid.
If you want to learn more about photometric stereo techniques, here are some great resources:
Self-translucency
Small rock from the Dolomites collection using Musou black fabrics.
Photogrammetry
Foliage is almost everything photogrammetry doesn’t want in a subject: thin, smooth and flexible. Yet there may still be a place for it.
Test case: Petasites Hybridus
Was primarily chosen since it was quite large and major vertical undulating shapes and some overlap.
Most of the time was spent on:
Processing each leaf individually. Partly due to size, and partly due to the accuracy needed when doing this kind of scanning.
Creating a clean alpha map for cut-out despite the bright LED background
Clean mesh cut-out and retopology on the final mesh before baking the textures
Most of these things are relatively automatable. Most of it will depend on how reliable your captures are. High consistency means reliable automated output, while low consistency means manual input. This is the crux of why photometric stereo is king.
Pain points:
How to ensure leaf doesn’t deform at all during the process
Some kind of spray?
How to get fully aligned subsurface (without syncing camera/LED)
Currently only 4 images are taken with the backlight on. This is enough for relatively accurate alignment, but minor offset will probably occur
Clean retopology has quite a few steps
Raw mesh/Color/SSS
02
H1
H2
03
Asset Assembly
Here is where things get interesting.
As 3D foliage has been evolving quite rapidly of the years, so has the assembly workflow. Proceduralism now leads the way forward, partly because of better tooling but also because of the complexity of foliage—not only from an artistic side, but also from a technical setup POV.
Proceduralism
There are many reasons to make foliage procedurally:
Much easier to get results, especially for beginners
Presets/recipes can be shared
Infinite variations and adaptations
Non-destructive
Easier to adapt asset to technical requirements such as wind
However proceduralism lacks control, and can become very complex and difficult to work with.
Modularity
This concept is becoming increasingly important as complexity scales.
Nature is repetition, yet it seems so unique because of the incomprehensible scale and complexity. Our job is to break down the complexity into as few modules as possible, as each one has its associated costs.
There is only so much variation you can notice, especially on the micro scale. This is why macro scale variation is the most difficult to nail down as repetition becomes much more noticeable, while combining micro and macro comes with many additional challenges. A case study of its own.
Handcraft
In the days of proceduralism and AI, saying this might seems like crazy talk, but it’s a reality that shouldn’t be ignored. One of the things that makes the difference is exactly the greasy manual work that happens in parts of the pipeline.
That said, this requires generally high skill and knowledge of foliage creation methods to be properly leveraged.
In the end, we believe that a strong foliage pipeline needs to have all 3 elements:
Proceduralism
Modularity
Handcraft
Breakdown
Species: Pinus halepensis
Needle trees are probably the worst case scenario for Nanite—and probably in general—because:
Needles get culled and tree loses density very quickly
High triangle counts that create very little occlusion culling
Image rendering instability
This used to not be a big deal back when we were using alpha cards, but since we are chasing complexity it isn’t an ideal solution anymore.
For Unreal, voxels was introduced to solve these problems, but to effectively be usable in a production you would need to adapt full Nanite Foliage setup, meaning this features usability is hampered.
Core modules
These are the individual meshes that will be used to build the hierarchies.
Hierarchies
Modules are broken down into hierarchies. For each hierarchy, a new root branch is added and a larger branch is built using the previous hierarchy. In other words, the branch gets bigger using mostly existing data.
H3
H4
Building the hierarchies
H5
Here we have to decide our approach. Do we generate procedurally, or do we dive in hands first.
In this case, the hierarchies were built manually as it gives the greatest amount of control. By semi-automating certain parts of the manual workflow, it doesn’t have to take that much time to do this.
A large part of this is admittedly experience through repetition, which helps your ability to minimize iterations. Inexperienced artists will start struggling here.
Depending on the trunk size, the end thickness diameter is made to fit either H4-5-6.
That way its easy to close the tree with a corresponding hierarchy module.
At this stage, the branches are not collapsed and everything is interchangeable. This gives you some power to make changes that will propagate through all of your already made branch modules.
Large scale structural changes can be tricky to make though, this is where doing it fully procedurally would make things much easier.
In this case we will focus on converting the tree to a alpha card setup
H6
The scanned trunk bases are extended, shaped and detailed. In the case of the Aleppo Pine, there are 3 distinct bark stages found throughout the tree. These are blended in depending on main trunk distance to the ground and branch thickness.
Leveraging modularity
Trunk base
Engine & Shaders
04
Part 2
05
Planning a project involving foliage? Let's have a chat!
+357 996 25 761 — 09-21 | GMT+2