Introduction
We are Aleyna and Francesco, and in this post, we present FG Sketches—a generative tool developed as part of the Artificial Intelligence in Architecture (AIA) module within the MaCAD Generative AI program. The objective of this project was to design a system capable of learning from the sketching language of Frank Gehry and translating that knowledge into an accessible platform for generating Gehry-inspired architectural sketches from basic image inputs.
Design Intent
Sketching is a foundational tool in the early stages of architectural design, serving as a medium for spatial exploration and ideation. Frank Gehry, in particular, is recognized for his use of rapid, gestural hand sketches as a means to investigate form. This project explores the possibility of embedding that visual spontaneity into a generative model—shifting the role of AI from rendering static visuals to enabling expressive, sketch-like interpretations.
The primary research question we addressed was: Can a generative system emulate the fluidity and abstraction characteristic of Gehry’s early design sketches?
Project Aim
- Model Development: To fine-tune a lightweight generative model capable of converting architectural image inputs into expressive, sketch-like outputs aligned with Gehry’s aesthetic vocabulary.
- Interface Design: To implement a user-friendly, interactive interface supporting conceptual ideation workflows in early design phases.
Workflow Overview
The FG Sketches pipeline operates in three core stages:
- Depth Map Generation: Input images are first converted into depth maps using ControlNet to extract geometric structure.
- Conditional Generation: These depth maps serve as conditioning inputs for Stable Diffusion, guiding the spatial consistency of the output.
- Style Transfer via LoRA: A custom LoRA (Low-Rank Adaptation) model, fine-tuned on Gehry’s sketches, applies the stylistic transformation to achieve a gestural, sketch-like quality.
This workflow allows the model to maintain architectural legibility while introducing the loose, dynamic linework characteristic of Gehry’s hand sketches.
LoRA Fine-Tuning
The fine-tuning process leveraged a curated dataset of Frank Gehry’s sketches, selected for their clear gestural qualities and minimal annotation noise. Each image was manually captioned with descriptors reflecting visual attributes such as curvature, scale, and line density.
LoRA fine-tuning was employed to ensure efficient training and focused style transfer. This approach preserves the fidelity of the input structure while adapting the output’s visual language to resemble Gehry’s expressive drawing style.
User Interface
A functional prototype was developed using Gradio, enabling real-time interaction. Users can upload architectural visuals—such as massing models, façade studies, or site images—and adjust key parameters including:
- Sketch intensity
- Number of inference steps
The output is generated within seconds and can be downloaded for further design development or presentation purposes.
Use Case Scenario
A typical application might involve a student preparing for a design review. By uploading a screenshot of a Rhino massing model, the user receives several stylized outputs that retain the original form while expressing it through dynamic sketch lines. This process accelerates visual iteration and supports early-stage abstraction without the time-intensive nature of manual drawing.
Results and Observations
The model consistently produces outputs with curved, fluid linework and a reduction in detail that favors silhouette and motion. Structural edges are typically preserved, while surface ornamentation is abstracted or omitted—reflecting the sketching ethos we aimed to replicate.
FG Sketches is not intended to supplant the role of the designer. Rather, it functions as a creative augmentation tool—enhancing generative workflows with the expressive clarity of hand-drawn ideation.
