One of TileBuddy’s most popular features is AI tile scanning — point your camera at a mahjong hand, snap a photo, and the app identifies every tile and calculates your faan automatically. But how does it actually work? Let’s pull back the curtain.

The Problem: Manual Tile Input Is Tedious

Before AI scanning, the only way to use a scoring app was to manually tap each tile on a virtual keyboard. For a 14-tile winning hand, that’s 14 taps at minimum, plus selecting the correct tile from a grid of 34 different tile types. It works, but it’s slow — especially when everyone’s waiting to start the next round.

AI scanning reduces this to a single photo.

The Technology Behind It

TileBuddy’s tile scanning uses Google’s Gemini API, a powerful multimodal AI that can understand images and identify objects within them.

How the Process Works

Capture — You take a photo of the mahjong tiles (or select one from your gallery)
Pre-processing — The app prepares the image for analysis, optimizing resolution and format
AI Recognition — The image is sent to the Gemini API, which identifies each individual tile
Tile Mapping — The recognized tiles are mapped to TileBuddy’s internal tile representation
Verification — Results are displayed for the user to confirm or adjust
Scoring — The confirmed tiles are run through TileBuddy’s faan calculation engine

The entire process takes just a few seconds.

Why Gemini?

We chose Google’s Gemini API for several reasons:

Multimodal Understanding

Gemini can process images and understand context in ways that traditional computer vision can’t. It doesn’t just see shapes and colors — it understands that it’s looking at mahjong tiles and can identify each one by its specific suit, number, and type.

Accuracy

Mahjong tiles have varied designs across different sets. Some are ornate with traditional artwork, others are simple and modern. Gemini handles this variation well because it understands the conceptual identity of each tile, not just pixel patterns.

Speed

The API returns results quickly enough to feel responsive. A 2-3 second wait is acceptable in the context of scoring a mahjong hand (compared to the minutes it might take to score manually).

Reliability

Cloud-based AI means TileBuddy benefits from ongoing model improvements without requiring app updates. As Gemini gets better at recognizing tiles, TileBuddy’s scanning improves automatically.

What the AI Sees

When you take a photo of mahjong tiles, the AI processes several visual elements:

Tile Face Design

Each tile type has a distinctive face:

Bamboo tiles — Sticks in varying quantities (1-bamboo is typically a bird)
Dot tiles — Circles arranged in patterns
Character tiles — Chinese numerals with the 萬 character
Wind tiles — Single Chinese characters (東南西北)
Dragon tiles — Distinctive colors/characters (中, 發, 白)

Tile Count

The AI counts how many tiles are present and flags if the count seems wrong (not 14, or not a valid hand size).

Tile Orientation

Tiles can be photographed at various angles and orientations. The AI handles rotated tiles, though face-up flat tiles give the best results.

Tips for Better Scanning

Lighting

Good, even lighting makes a huge difference. Avoid:

Harsh shadows across tile faces
Direct overhead light causing glare
Very dim environments

Natural daylight or soft indoor lighting works best.

Arrangement

Lay tiles face-up on a flat surface
Space tiles apart slightly so edges don’t overlap
Use a contrasting background (dark tiles on light surface or vice versa)
Revealed sets and concealed sets can be photographed together

Camera Position

Hold the phone directly above the tiles for the best angle
Make sure all tiles are within the frame
Avoid extreme angles that cause perspective distortion
Landscape orientation often captures wide tile spreads better

Tile Cleanliness

Dirty or worn tiles with faded markings are harder to scan. Most modern tile sets scan well, but very old or heavily used sets may require some manual correction.

Accuracy and Edge Cases

What It Handles Well

Standard modern tile sets with clear markings
Tile sets with different artistic styles
Hands with flowers/seasons included
Kong tiles (groups of four)

Where It May Struggle

Very worn or faded tiles
Tiles partially obscured by other tiles
Extreme lighting conditions (too dark or too bright)
Unusual or highly stylized tile designs
Blurry photos

The Hybrid Approach

This is why TileBuddy always shows you the recognition results before calculating faan. If the AI misidentifies a tile (say, it reads a 6-Bamboo as an 8-Bamboo), you simply tap the incorrect tile and manually select the right one. In practice, most scans require zero or one corrections.

Privacy and Data

A natural question: what happens to the photos you take?

Images are sent to the Gemini API for processing
TileBuddy does not store your photos on external servers
The AI processing happens in real-time and images are not retained by the API for training
You can use manual input if you prefer not to use the camera at all

The Future of AI in Mahjong

AI tile scanning is just the beginning. Potential future applications include:

Real-time hand advice — AI suggesting optimal discards based on your current hand
Post-game analysis — Reviewing your play decisions against optimal strategy
Automatic round tracking — Using the camera to detect wins, discards, and game state changes
Voice-activated scoring — Calling out tiles verbally instead of scanning

For now, tile scanning solves the most immediate pain point: getting from “tiles on the table” to “faan count” as quickly as possible.

FAQ

Does AI scanning work offline?

No. The AI scanning feature requires an internet connection because it uses the Gemini API for image recognition. Manual tile input works fully offline.

Is my data safe when using AI scanning?

Yes. Photos are processed in real-time through the Gemini API and are not stored or used for training. TileBuddy doesn’t retain your images on any external server.

How accurate is the tile recognition?

In normal lighting conditions with modern tile sets, accuracy is above 95%. Most errors involve visually similar tiles (like 6-Bamboo and 8-Bamboo) and are easily corrected with a single tap.

Can I scan tiles that are still in the wall or in my hand?

The scanner works best with tiles laid flat and face-up. You can photograph tiles held in your hand, but accuracy will be lower due to the angle and potential partial obscuring.

See the magic for yourself. Download TileBuddy for free on the App Store and try AI tile scanning on your next winning hand.

How AI Tile Scanning Works in TileBuddy