Texture Synthesis Algorithms and Applications - From Patch-Based to Deep Learning

What Is Texture Synthesis - Core Concepts and Applications

Texture synthesis is the process of automatically generating a large texture from a small sample image while preserving its visual characteristics. It is widely used in game development, film VFX, architectural visualization, and 3D modeling to create seamless, arbitrarily-sized surface materials.

The problem texture synthesis solves: Real-world textures captured by cameras have limited resolution and size. Simple tiling creates visible seams and repetition artifacts that break visual immersion. Texture synthesis generates new content that statistically matches the input while avoiding obvious repetition patterns.

Texture classification:

• Stochastic textures: Sand, clouds, noise - low regularity, relatively easy to synthesize with statistical methods
• Structural textures: Bricks, tiles, woven fabric - clear patterns where structure preservation is critical
• Near-regular textures: Wood grain, stone walls - partial regularity, most challenging to synthesize

Quality evaluation relies on perceptual similarity metrics including SSIM, FID (Frechet Inception Distance), and subjective human assessment. With growing demand for 4K+ textures in modern game engines and film production, efficient synthesis algorithms have become increasingly important for production pipelines.

Pixel-Based Methods - The Efros-Leung Algorithm

Pixel-based texture synthesis, pioneered by Efros and Leung in 1999, generates output textures one pixel at a time by searching the input for the most similar neighborhood pattern. This foundational approach established the non-parametric framework for texture synthesis.

Algorithm steps:

• Extract the neighborhood window (e.g., 11x11) around the current unfilled pixel in the output
• Slide a same-sized window across the input texture, computing SSD (Sum of Squared Differences) at each position
• Select the best matching patch and copy its center pixel value to the output
• Process all pixels in raster-scan or onion-peel order

Neighborhood window size effects: Small windows (5x5) capture only local features, causing global structure breakdown. Large windows (23x23+) preserve structure but increase computation to O(N²) and tend toward simple copying. Practical sizes range from 9x9 to 15x15 depending on texture scale.

Computational cost: Generating a 256x256 output requires exhaustive search across the input for each of 65,536 pixels. With a 128x128 input, this means approximately 1 billion comparison operations. Acceleration techniques include ANN (Approximate Nearest Neighbor) search and tree-structured vector quantization, but real-time processing remains impractical. Typical generation time is several minutes to tens of minutes for 256x256 output.

Patch-Based Methods - Image Quilting and GraphCut

Patch-based methods synthesize textures by placing rectangular blocks rather than individual pixels, dramatically improving both quality and speed. Image Quilting (Efros-Freeman, 2001) and GraphCut Textures (2003) are the landmark approaches in this category.

Image Quilting algorithm:

• Divide the output into a grid with overlapping regions (e.g., patch size 36x36, overlap 6 pixels)
• For each grid position, search the input for patches whose overlap region SSD falls below a threshold
• Compute the minimum error boundary cut through the overlap region using dynamic programming
• Stitch patches along the optimal boundary to minimize visible seams

GraphCut improvement: GraphCut Textures formulates the optimal boundary between patches as a graph cut (max-flow/min-cut) problem. Unlike Image Quilting's 1D boundary, GraphCut finds 2D optimal boundaries, producing more natural results especially for structured textures. The computational overhead is higher but quality gains are significant for regular patterns.

Practical parameter settings: Patch size should be 1.5-2x the characteristic scale of the texture features. For brick walls, this means including at least one full brick; for wood grain, approximately twice the grain repetition period. Overlap should be 1/6 to 1/4 of patch size - too large increases computation, too small reveals seams. These methods can generate 512x512 textures in under 1 second on modern CPUs.

Statistical Methods - Gram Matrices and Neural Style Transfer

The neural style transfer approach proposed by Gatys et al. in 2015 represents texture statistics using Gram matrices of CNN feature maps. This parametric texture model captures complex statistical properties that traditional methods cannot express, enabling high-quality synthesis of stochastic textures.

Gram matrix texture representation: Feature maps from VGG-19 convolutional layers (size C×H×W) are reshaped to C×(H*W) matrices. The inner product produces a C×C Gram matrix that captures inter-channel correlations, representing texture statistics in a position-independent manner. Multiple layers capture features at different scales.

Synthesis procedure:

• Start from a random noise image
• Compute style loss as the difference between Gram matrices of input texture and generated image at each layer
• Iteratively update generated image pixels using L-BFGS or Adam optimizer
• Typically converges in 300-500 iterations

Advantages and limitations: This method excels at stochastic textures (clouds, water surfaces, gravel) but struggles with structural textures (bricks, lattices) because Gram matrices discard spatial position information, making periodic pattern reproduction difficult. Computation cost is high - approximately 30 seconds on GPU for 512x512 generation. Feed-forward networks (Ulyanov et al., 2016) reduce inference to milliseconds by training a generator network to directly produce textures matching target Gram statistics.

GAN-Based Texture Synthesis - High Quality at High Speed

GAN-based texture synthesis represents the state of the art, producing high-quality results at near-real-time speeds. Key approaches include PSGAN, SinGAN, and StyleGAN-based methods that leverage adversarial training for photorealistic texture generation.

SinGAN (2019): Trains a multi-scale GAN from a single image to generate textures of arbitrary size. The pyramid architecture generates from coarse to fine scales, preserving both global structure and local detail. Training requires only a single image but takes approximately 30 minutes per image on GPU. Once trained, generation is fast at under 100ms.

StyleGAN-based methods: Leverage StyleGAN2 architecture to generate textures from latent space. Style vector manipulation enables continuous control over texture attributes such as color tone, roughness, and directionality. Transfer learning from large texture datasets with few-shot fine-tuning is the practical approach for production use.

Practical comparison:

• Quality: GAN-based > Gram matrix > Patch-based > Pixel-based (for structural textures)
• Inference speed: GAN-based (10ms) > Patch-based (1s) > Gram matrix (30s) > Pixel-based (minutes)
• Controllability: GAN-based offers high controllability through latent space manipulation
• Training cost: GANs require hours to days of pre-training on GPU clusters

In industry, game engines like Unreal Engine 5 and Unity are beginning to integrate real-time texture synthesis, with Nanite-compatible dynamic texture generation gaining significant attention for next-generation rendering pipelines.

Specialized books on CG texture techniques are also helpful

Practical Texture Synthesis - Tools and Workflows

This section covers concrete tools, libraries, and workflows for applying texture synthesis in production environments, with recommendations for different use cases and quality requirements.

Open-source libraries:

• OpenCV: cv2.inpaint() provides texture-based inpainting using Telea and Navier-Stokes methods
• scikit-image: The skimage.restoration module offers patch-based synthesis capabilities
• PyTorch: Gram matrix synthesis implementable with torchvision.models.vgg19, GPU-accelerated to generate 512x512 in ~30 seconds
• Substance Designer: Industry-standard procedural texture tool with node-based non-destructive workflow

Creating seamless textures:

• Cross-fade blending at image edges with overlap regions
• Image Quilting with periodic boundary conditions for seamless output
• Frequency domain phase adjustment to ensure periodic continuity

Quality verification: Tile the result in 2x2 or 3x3 grids to check for visible seams. Compare histograms between input and output to verify color distribution consistency. For structural textures, compare FFT power spectra to confirm periodicity preservation. FID scores below 50 generally indicate high-quality synthesis results suitable for production use.

Performance optimization: For 4K texture generation, multi-scale approaches are effective. Start with 256x256 coarse synthesis, then progressively upsample while adding detail at each level. This achieves 5-10x speedup compared to direct 4K generation while maintaining visual quality.