cot_reports

Table of Content

• cot_reports
• Installation
• Requirements
• Usage
• Introduction to the COT reports
  • Data release
  • COT report types
  • Format of the reports
  • Classification methodology Legacy reports
  • Data aggregation
  • Use cases of the COT reports
• Sources
• Roadmap
• License

cot_reports

cot_reports is a Python library for fetching the Commitments of Trader reports of the Commodity Futures Trading Commission (CFTC). The following Commitments of Trader reports are supported: Legacy Futures-only, Legacy Futures-and-Options Combined, Supplemental Futures-and-Options Combined, Disaggregated Futures-only, Disaggregated Futures-and-Options Combined, Traders in Financial Futures (TFF) Futures-only and Traders in Financial Futures (TFF) Futures-and-Options Combined.

Please note: The functionality of the libraries' functions may interrupt in case any changes in the publication occur or in case the website is not available.

Installation

Use the package manager pip to install cot_reports

pip install cot_reports  

Or install the library through the author's Github repository

pip install git+https://github.com/NDelventhal/cot_reports

Requirements

The following libraries are required:

• pandas
• requests
• beautifulsoup4

These libraries can be installed via the package manager pip.

pip install pandas requests beautifulsoup4

Usage

Available report types with the corresponding 'cot_report_type' name used in the library:

report	'cot_report_type'
Legacy Report Futures-only	'legacy_fut'
Legacy Report Futures-and-Options Combined	'legacy_futopt'
Supplemental Report Futures-and-Options combined (short format)	'supplemental_futopt'
Disaggregated Report Futures-only	'disaggregated_fut'
Disaggregated Report Futures-and-Options Combined	'disaggregated_futopt'
Traders in Financial Futures (TFF)	'traders_in_financial_futures_fut'
Traders in Financial Futures (TFF) Futures-and-Options Combined	'traders_in_financial_futures_futopt'

Usage examples:

• cot_hist() downloads the compressed bulk files for the specified report (cot_report_type) starting from 1986 (depending on the report type) up to 2016 and returns the data in a dataframe.

• cot_year() downloads historical single year data for the specified report (cot_report_type) and returns the data in a dataframe (cot_report_type). If the current year is specified, the latest published data is fetched.

• cot_all() downloads the complete available data, including the latest, of the specified report and returns the data in a dataframe (cot_report_type).

• cot_all_reports() downloads all available historical information of the available COT reports and returns the data as seven dataframes.

import cot_reports as cot

# Example: cot_hist()
df = cot.cot_hist(cot_report_type= 'traders_in_financial_futures_futopt')
# cot_hist() downloads the historical bulk file for the specified report type, in this example the Traders in Financial Futures Futures-and-Options Combined report. Returns the data as dataframe.

# Example: cot_year()
df = cot.cot_year(year = 2020, cot_report_type = 'traders_in_financial_futures_fut')
# cot_year() downloads the single year file of the specified report type and year. Returns the data as dataframe.

# Example for collecting data of a few years, here from 2017 to 2020, of a specified report:
df = pd.DataFrame()
begin_year = 2017
end_year = 2020
for i in range(begin_year, end_year + 1):
    single_year = pd.DataFrame(cot.cot_year(i, cot_report_type='legacy_futopt')) 
    df = df.append(single_year, ignore_index=True)

# Example: cot_all()
df = cot.cot_all(cot_report_type='legacy_fut')
# cot_all() downloads the historical bulk file and all remaining single year files of the specified report type.  Returns the data as dataframe.

Introduction to the COT reports

To promote its goals of integrity, resilience and vibrancy of the U.S. derivatives markets through regulation, the U.S. Commodity Futures Trading Commission (CFTC) relies on collected data to conduct its functions. The CFTC's functions:

• market oversight
• monitoring for liquidity and systemic risk
• oversight of market participants
• regulatory compliance
• enforcement of the Commodity Exchange Act (CEA)

In order to conduct these functions, data is seen as a critical asset to the CFTC, which states to have been one of the first federal agencies who implemented a data management organization in 2011. The CFTC is not solely collecting but also publishing a variety of reports on the derivative markets under its jurisdiction. These reports include the Commitment of Traders (COT) reports.

Through the publication of the COT reports the CFTC claims 'to help the public to understand market dynamics.'

The COT reports provide a breakdown of each Tuesday’s open interest for futures and options on futures markets in which 20 or more traders hold positions equal to or above the reporting levels established by the CFTC.

The data is reported to the CFTC by firms, such as:

• futures commission merchants (FCMs)
• clearing members
• foreign brokers
• exchanges

The actual trader classification is based on the predominant business purpose, which is self-reported by traders on the CFTC Form 401. The classification of a trader can vary for different commodities/markets, if the reported business purposes differ. However, the classifications are subject to review and corrections by the CFTC.

Data release

The data, which is generally released each Friday at 3:30 pm Eastern time, comes with a lag of three days, as the reported data is typically from previous Tuesday (close). According to the commission the firms report to the CFTC on Wednesday morning. Next, the provided information is then subject to corrections and verifications by the CFTC prior to the release on Friday.

COT report types

The types of the COT reports:

1. Legacy

   • physical contracts & non-physical contracts
   • reportable position classifications:
     • non-commercial
     • commercial traders
   • available as Futures-only and Futures-and-Options Combined reports
   • available from 1986 and 1995
   • available in long and short format

2. Supplemental

   • selected agricultural commodity contracts
   • Supplemental reports break down the reportable open interest positions into three trader classifications
   • reportable position classifications:
     • non-commercial
     • commercial
     • index traders
   • available as Futures-and-Options Combined report
   • available from 2006
   • available in short format
   • Why was this report established? With the publication of the Supplemental COT report the CFTC has reacted on the rise of index traders. The commission started to publish the Supplemental report starting in 2006 with a further trader categorization of index traders for selected agricultural products to prevent a blending of traditional commercial positions and also of traditional non-commercial position with index funds positions. With the newly introduced classification of index traders misinterpretation related to export activity buying or selling end-user are avoided, by differentiating between traditional and non-traditional hedgers as well as between traditional non-commercial and index funds open interest. In general from the non-commercial category managed funds, pension funds and other institutional investors that are generally seeking exposure to commodity prices as an asset class in an unleveraged and passively-managed manner through standardized commodity indices are drawn into the index trader categorization. From the commercial category entities, whose positions predominantly reflect hedging of OTC transactions through commodity indices, are drawn into the index trader categorization. The Supplemental reports solved a disadvantage with respect to the usefulness of the information and serves the original intent in publishing the COT data better by enhancing the transparency.

3. Disaggregated

   • data on physical contracts, such as contracts on agricultural products, petroleum products, natural gas products etc.
   • reportable position classifications:
     • Producer/Merchant/Processor/User
     • Swap Dealers
     • Managed Money
     • Other Reportable
   • available as Futures-only and Futures-and-Options Combined reports
   • available from 2006
   • available in long and short format

Format of the reports

The COT reports are either available in the short format, the long format or both - as specified above. Within the long format, aside from the additional listing of the concentrations of positions held by the largest four and largest eight traders, the data is grouped by crop year, where appropriate. Except, where not available (cot report type: Supplemental), the library accesses the long format.

Data aggregation

Aggregations occur in the reports not only in the classification of the trader, but also by merging future positions with differing expiration dates. Due to this handling continuous sentiment data on a product, referred to as market by the CFTC and within the COT reports, can be derived. As the data is highly aggregated, any results derived from the data should be interpreted with caution.

Classification Methodology Legacy reports

Not only does the Legacy Futures-only COT report offer the greatest available historical data dating back to 1986, the Legacy reports also contain the most markets in comparison to the other COT report types. Unlike the others, the Legacy reports contain data on physical delivery contracts and non-physical delivery (financial) contracts. While the Supplemental and the Disaggregated reports are limited to physical delivery contracts, the Traders in Financial Futures reports are limited to non-physical financial contracts.

Classification Methodology: Legacy reports

The positions of the market participants in the Legacy reports are broken down into:

• Commercials

  • market participants who use futures contracts for hedging as defined in CFTC Regulation 1.3, 17 CFR 1.3(z) and which hold positions equal to or above the reporting levels established by the CFTC
  • CFTC Form 40 declaration: the trader is engaged in business activities, which can be hedged by the use of futures (or options)
  • Example: A gold mining company, such as Goldcorp or Barrick Gold, would be classified as commercials in gold futures, as well as other merchants, processors or users buying and selling Gold
  • using descriptions of the other reports I derive the commercials to contain primarily producer/merchant/processor/user and swap dealers

• Non-commercials

  • market participant whose positions equal or are above the reporting levels of by the CFTC and are not considered as commercials
  • As a consequence this group is derived to have speculative interest and is commonly considered as speculators
  • using descriptions of the other reports I derive