TSV in Pandas: A How-To Guide. The Correct Way of Loading and Writing… | by Michał Marcińczuk, Ph.D. | Oct, 2024

The Correct Way of Loading and Writing TSV Files with Pandas

TSV is a widely used format for storing tabular data, but it can be confusing when working with textual data and the Pandas library. Two factors cause the confusion:

1. TSV is very similar to CSV (a well-known format for storing data), but it is not the same.
2. Pandas default settings are not compatible with the TSV format.

In the story, I briefly discuss the source of confusion and present the best way to handle the TSV format using the Pandas library.

TSV [1] is a simple file format similar to CSV. However, there are several important differences:

1. It uses tabs to separate the fields.
2. It does not allow some characters, that is, line feed (\n), tabs (\t) and carriage (\r) returns within fields.
3. There are no quotations of the fields nor escapes of special characters [2] (at least for the original format).

Point 2 is problematic when dealing with text fields, as they might contain the forbidden characters. The suggested way to deal with the forbidden characters is to replace them with arbitrary texts, like…