非结构化文件

这个笔记本介绍了如何使用Unstructured包来加载各种类型的文件。Unstructured目前支持加载文本文件、PowerPoint文件、HTML文件、PDF文件、图像文件等等。

# # Install package
!pip install "unstructured[local-inference]"
!pip install layoutparser[layoutmodels,tesseract]

# # Install other dependencies
# # https://github.com/Unstructured-IO/unstructured/blob/main/docs/source/installing.rst
# !brew install libmagic
# !brew install poppler
# !brew install tesseract
# # If parsing xml / html documents:
# !brew install libxml2
# !brew install libxslt

# import nltk
# nltk.download('punkt')

from langchain.document_loaders import UnstructuredFileLoader

loader = UnstructuredFileLoader("./example_data/state_of_the_union.txt")

docs = loader.load()

docs[0].page_content[:400]

'Madam Speaker, Madam Vice President, our First Lady and Second Gentleman. Members of Congress and the Cabinet. Justices of the Supreme Court. My fellow Americans.\n\nLast year COVID-19 kept us apart. This year we are finally together again.\n\nTonight, we meet as Democrats Republicans and Independents. But most importantly as Americans.\n\nWith a duty to one another to the American people to the Constit'

保留元素

在底层，Unstructured为不同的文本块创建不同的"元素"。默认情况下，我们将它们组合在一起，但您可以通过指定mode="elements"来轻松保留这种分离。

loader = UnstructuredFileLoader("./example_data/state_of_the_union.txt", mode="elements")

docs = loader.load()

docs[:5]

[Document(page_content='Madam Speaker, Madam Vice President, our First Lady and Second Gentleman. Members of Congress and the Cabinet. Justices of the Supreme Court. My fellow Americans.', lookup_str='', metadata={'source': '../../state_of_the_union.txt'}, lookup_index=0), Document(page_content='Last year COVID-19 kept us apart. This year we are finally together again.', lookup_str='', metadata={'source': '../../state_of_the_union.txt'}, lookup_index=0), Document(page_content='Tonight, we meet as Democrats Republicans and Independents. But most importantly as Americans.', lookup_str='', metadata={'source': '../../state_of_the_union.txt'}, lookup_index=0), Document(page_content='With a duty to one another to the American people to the Constitution.', lookup_str='', metadata={'source': '../../state_of_the_union.txt'}, lookup_index=0), Document(page_content='And with an unwavering resolve that freedom will always triumph over tyranny.', lookup_str='', metadata={'source': '../../state_of_the_union.txt'}, lookup_index=0)]

定义分割策略

Unstructured文档加载器允许用户传递一个strategy参数，让unstructured知道如何对文档进行分割。目前支持的策略有"hi_res"（默认值）和"fast"。高分辨率的分割策略更准确，但处理时间更长。快速策略能更快地对文档进行分割，但会牺牲准确性。并非所有文档类型都具有单独的高分辨率和快速分割策略。对于这些文档类型，strategy关键字参数将被忽略。在某些情况下，如果存在依赖关系缺失（例如文档分割的模型），高分辨率策略将回退到快速策略。以下是如何将策略应用于UnstructuredFileLoader的示例：

from langchain.document_loaders import UnstructuredFileLoader

loader = UnstructuredFileLoader("layout-parser-paper-fast.pdf", strategy="fast", mode="elements")

docs = loader.load()

docs[:5]

[Document(page_content='1', lookup_str='', metadata={'source': 'layout-parser-paper-fast.pdf', 'filename': 'layout-parser-paper-fast.pdf', 'page_number': 1, 'category': 'UncategorizedText'}, lookup_index=0), Document(page_content='2', lookup_str='', metadata={'source': 'layout-parser-paper-fast.pdf', 'filename': 'layout-parser-paper-fast.pdf', 'page_number': 1, 'category': 'UncategorizedText'}, lookup_index=0), Document(page_content='0', lookup_str='', metadata={'source': 'layout-parser-paper-fast.pdf', 'filename': 'layout-parser-paper-fast.pdf', 'page_number': 1, 'category': 'UncategorizedText'}, lookup_index=0), Document(page_content='2', lookup_str='', metadata={'source': 'layout-parser-paper-fast.pdf', 'filename': 'layout-parser-paper-fast.pdf', 'page_number': 1, 'category': 'UncategorizedText'}, lookup_index=0), Document(page_content='n', lookup_str='', metadata={'source': 'layout-parser-paper-fast.pdf', 'filename': 'layout-parser-paper-fast.pdf', 'page_number': 1, 'category': 'Title'}, lookup_index=0)]

PDF示例

处理PDF文档的方式与上述相同。Unstructured会检测文件类型并提取相同类型的元素。

!wget  https://raw.githubusercontent.com/Unstructured-IO/unstructured/main/example-docs/layout-parser-paper.pdf -P "../../"

loader = UnstructuredFileLoader("./example_data/layout-parser-paper.pdf", mode="elements")

docs = loader.load()

docs[:5]

[Document(page_content='LayoutParser : A Unified Toolkit for Deep Learning Based Document Image Analysis', lookup_str='', metadata={'source': '../../layout-parser-paper.pdf'}, lookup_index=0), Document(page_content='Zejiang Shen 1 ( (ea)\n ), Ruochen Zhang 2 , Melissa Dell 3 , Benjamin Charles Germain Lee 4 , Jacob Carlson 3 , and Weining Li 5', lookup_str='', metadata={'source': '../../layout-parser-paper.pdf'}, lookup_index=0), Document(page_content='Allen Institute for AI shannons@allenai.org', lookup_str='', metadata={'source': '../../layout-parser-paper.pdf'}, lookup_index=0), Document(page_content='Brown University ruochen zhang@brown.edu', lookup_str='', metadata={'source': '../../layout-parser-paper.pdf'}, lookup_index=0), Document(page_content='Harvard University { melissadell,jacob carlson } @fas.harvard.edu', lookup_str='', metadata={'source': '../../layout-parser-paper.pdf'}, lookup_index=0)]

Unstructured API

如果您希望尽量减少设置步骤，只需运行pip install unstructured并使用UnstructuredAPIFileLoader或UnstructuredAPIFileIOLoader即可。这将使用托管的Unstructured API处理您的文档。请注意，目前（截至2023年5月11日），Unstructured API是开放的，但很快将需要一个API密钥。一旦可用，Unstructured文档页面将提供有关如何生成API密钥的说明。如果您想自行托管Unstructured API或在本地运行，请参考这里的说明。

from langchain.document_loaders import UnstructuredAPIFileLoader

filenames = ["example_data/fake.docx", "example_data/fake-email.eml"]

loader = UnstructuredAPIFileLoader(
    file_path=filenames[0],
    api_key="FAKE_API_KEY",
)

docs = loader.load()
docs[0]

Document(page_content='Lorem ipsum dolor sit amet.', metadata={'source': 'example_data/fake.docx'})

您还可以使用UnstructuredAPIFileLoader通过单个API批处理多个文件。

loader = UnstructuredAPIFileLoader(
    file_path=filenames,
    api_key="FAKE_API_KEY",
)

docs = loader.load()
docs[0]

Document(page_content='Lorem ipsum dolor sit amet.\n\nThis is a test email to use for unit tests.\n\nImportant points:\n\nRoses are red\n\nViolets are blue', metadata={'source': ['example_data/fake.docx', 'example_data/fake-email.eml']})