Imagine transforming any 3D model into an expressive, hand-rendered-style watercolor sketch, a visualization that breathes life into architectural concepts.
Pleinar: the interface that transforms 3D models into captivating, multiview, generative watercolor sketches, inspired by the brilliant artistry of Liz Steel.
__________________________________________________________________________________________________________________________________________________________________________
__________________________________________________________________________________________________________________________________________________________________________
The LoRA – training DATASET
We trained this LoRA on the work of artist Liz Steel. The dataset contains 74 images spanning landscape, architecture, and everyday objects.
The LoRA – training PROCESS
The model reached its capacity for learning very quickly. While our training loss shows no clear convergence, our tests showed that the image generation pipeline works much better with the LoRA than without. So while we could have fine tuned the LoRA with a higher learning rate or further addressed the noisy gradient, we believe that the inference quality is good for our aim.
__________________________________________________________________________________________________________________________________________________________________________
The INTERFACE – System Diagram
Pleinar visualizes a 3D model, allowing the user to adjust the initial view parameters before generating eight snapshots of the model at different angles, from which the user can select a preferred perspective to be processed through SDXL.
The image then undergoes canny edge detection for edge structural integrity and Scribble ControlNet for artistic flexibility, followed by LoRA-specifically trained as described in the previous slides. Multiple sketches variations per each chosen perspective are generated, that the user is allowed to save and download.
The INTERFACE – Practical example
Here an applicative example, going from the extracted 3d model view to the watercolour sketchy view of the model.
MAIN STEPS:
1. The user uploads a 3d model, visualized in Pleinair using pyrender.
2. The user is allowed to adjust the initial view parameters such as the camera’s vertical tilt and zoom level.
3. The model is then rotated in eight snapshots, one every 45 degrees, so the user can pick the most compelling perspective to process through SDXL.
4. Once a perspective is selected, the image goes through two key edge-detection processes: canny edge detection and scribble controlnet.
5. Here’s also where the LoRA comes in, as previously described, enhancing the stylistic output.
6. From each chosen perspective, three unique watercolor sketches are generated, from which the user can select and save their favorites.
7. Saved images are stored in the GALLERY tab, where the users can select those they would like to download.
