← Back to blog
tilebuddy ai technology

How AI Tile Scanning Works in TileBuddy

Wesley Ng ·

One of TileBuddy’s most popular features is AI tile scanning: point your camera at a mahjong hand, snap a photo, and the app identifies every tile and calculates your faan automatically. But how does it actually work?

The Problem: Manual Tile Input Is Tedious

Before AI scanning, the only way to use a scoring app was to manually tap each tile on a virtual keyboard. For a 14-tile winning hand, that’s 14 taps at minimum, plus selecting the correct tile from a grid of 34 different tile types. It works, but it’s slow, especially when everyone’s waiting to start the next round.

AI scanning reduces this to a single photo.

The Technology Behind It

TileBuddy’s tile scanning uses Google’s Gemini API, a powerful multimodal AI that can understand images and identify objects within them.

How the Process Works

You take a photo of the mahjong tiles (or select one from your gallery), and the app prepares the image for analysis, optimizing resolution and format. From there it’s sent to the Gemini API, which identifies each individual tile and maps the results to TileBuddy’s internal tile representation. Those results are displayed for you to confirm or adjust, and once confirmed, the tiles run through TileBuddy’s faan calculation engine.

The entire process takes just a few seconds.

Why Gemini?

We chose Google’s Gemini API for a few reasons. It doesn’t just see shapes and colors: it understands that it’s looking at mahjong tiles and can identify each one by its specific suit, number, and type, handling the wide variation across tile sets (ornate traditional artwork versus simple modern designs) by grasping the conceptual identity of each tile rather than just matching pixel patterns. It’s also fast: a 2-3 second wait feels responsive compared to the minutes manual scoring can take. And because it’s cloud-based, TileBuddy benefits from ongoing model improvements without requiring app updates, so scanning gets better automatically as Gemini does.

What the AI Sees

When you take a photo of mahjong tiles, the AI processes several visual elements:

Tile Face Design

Each tile type has a distinctive face:

  • Bamboo tiles: Sticks in varying quantities (1-bamboo is typically a bird)
  • Dot tiles: Circles arranged in patterns
  • Character tiles: Chinese numerals with the 萬 character
  • Wind tiles: Single Chinese characters (東南西北)
  • Dragon tiles: Distinctive colors/characters (中, 發, 白)

Tile Count

The AI counts how many tiles are present and flags if the count seems wrong (not 14, or not a valid hand size).

Tile Orientation

Tiles can be photographed at various angles and orientations. The AI handles rotated tiles, though face-up flat tiles give the best results.

Tips for Better Scanning

Lighting

Good, even lighting makes a huge difference. Avoid:

  • Harsh shadows across tile faces
  • Direct overhead light causing glare
  • Very dim environments

Natural daylight or soft indoor lighting works best.

Arrangement

  • Lay tiles face-up on a flat surface
  • Space tiles apart slightly so edges don’t overlap
  • Use a contrasting background (dark tiles on light surface or vice versa)
  • Revealed sets and concealed sets can be photographed together

Camera Position

  • Hold the phone directly above the tiles for the best angle
  • Make sure all tiles are within the frame
  • Avoid extreme angles that cause perspective distortion
  • Landscape orientation often captures wide tile spreads better

Tile Cleanliness

Dirty or worn tiles with faded markings are harder to scan. Most modern tile sets scan well, but very old or heavily used sets may require some manual correction.

Accuracy and Edge Cases

What It Handles Well

  • Standard modern tile sets with clear markings
  • Tile sets with different artistic styles
  • Hands with flowers/seasons included
  • Kong tiles (groups of four)

Where It May Struggle

  • Very worn or faded tiles
  • Tiles partially obscured by other tiles
  • Extreme lighting conditions (too dark or too bright)
  • Unusual or highly stylized tile designs
  • Blurry photos

The Hybrid Approach

This is why TileBuddy always shows you the recognition results before calculating faan. If the AI misidentifies a tile (say, it reads a 6-Bamboo as an 8-Bamboo), you simply tap the incorrect tile and manually select the right one. In practice, most scans require zero or one corrections.

Privacy and Data

A natural question: what happens to the photos you take?

  • Images are sent to the Gemini API for processing
  • TileBuddy does not store your photos on external servers
  • The AI processing happens in real-time and images are not retained by the API for training
  • You can use manual input if you prefer not to use the camera at all

Tile scanning solves the most immediate pain point: getting from “tiles on the table” to “faan count” as quickly as possible.

FAQ

Does AI scanning work offline?

No. The AI scanning feature requires an internet connection because it uses the Gemini API for image recognition. Manual tile input works fully offline.

Is my data safe when using AI scanning?

Yes. Photos are processed in real-time through the Gemini API and are not stored or used for training. TileBuddy doesn’t retain your images on any external server.

How accurate is the tile recognition?

In normal lighting conditions with modern tile sets, accuracy is above 95%. Most errors involve visually similar tiles (like 6-Bamboo and 8-Bamboo) and are easily corrected with a single tap.

Can I scan tiles that are still in the wall or in my hand?

The scanner works best with tiles laid flat and face-up. You can photograph tiles held in your hand, but accuracy will be lower due to the angle and potential partial obscuring.

See the magic for yourself. Download TileBuddy for free on the App Store and try AI tile scanning on your next winning hand.