🎨SketchDeco:

Decorating B&W Sketchs with Colour

Chaitat Utintu*, Pinaki Nath Chowdhury, Aneeshan Sain, Subhadeep Koley, Ayan Kumar Bhumia, Yi-Zhe Song
*Interned with SketchX
SketchX, CVSSP, University of Surrey
Paper arXiv Code
Demo Image.

SketchDeco turns free-hand sketch, masks, and colour palettes into realistic image without user-defined text prompts.

Abstract

This paper introduces a novel approach to sketch colourisation, inspired by the universal childhood activity of colouring and its professional applications in design and storyboarding. Striking a balance between precision and convenience, our method utilises region masks and colour palettes to allow intuitive user control, steering clear of the meticulousness of manual colour assignments or the limitations of textual prompts.

By strategically combining ControlNet and staged generation, incorporating Stable Diffusion v1.5, and leveraging BLIP-2 text prompts, our methodology facilitates faithful image generation and user-directed colourisation. Addressing challenges of local and global consistency, we employ inventive solutions such as an inversion scheme, guided sampling, and a self-attention mechanism with a scaling factor. The resulting tool is not only fast and training-free but also compatible with consumer-grade Nvidia RTX 4090 Super GPUs, making it a valuable asset for both creative professionals and enthusiasts in various fields.

Architecture

Model Image.

Our training-free sketch colourisation pipeline.

Results

Model Image.

Comparison with SOTA diffusion-based approaches. Our training-free colourisation approach outperforms state-of-the-art (SOTA) models, DiffBlender and DiSS, in various aspects including colour vividness, colour harmonisation, fidelity to the original sketch, and overall realistic look.

Model Image.

Model Image.

Model Image.

Model Image.

Model Image.

Model Image.

Qualitative evaluation with random sketches. The local sketch colourisation pipeline showcased notable adaptability, allowing users to incorporate three conditional inputs: a variety of sketches, hand-drawn masks, and preferred colour palettes. Consequently, our approach yields precise outcomes tailored to user-defined specifications. It is non-trivial to mention that these sketch images are randomly sourced from www.freepik.com with search keywords: “black-and-white [class] sketch”.

BibTeX

@article{utintu2024sketchdeco,
  author    = {Chaitat Utintu and Pinaki Nath Chowdhury and Aneeshan Sain and Subhadeep Koley and Ayan Kumar Bhunia and Yi-Zhe Song},
  title     = {SketchDeco: Decorating B&W Sketches with Colour},
  journal   = {arXiv preprint arXiv:2405.18716},
  year      = {2024},
}