spaCy

spaCyopen in new window 是一个用于高级自然语言处理的开源软件库，使用 Python 和 Cython 编程语言编写。

除了 NLTK，另一个选择是使用 spaCy 标记器open in new window。

文本如何进行分割：使用 spaCy 标记器进行分割。
分块大小的测量方式：按字符数量进行测量。

#!pip install spacy

# This is a long document we can split up.
with open('../../../state_of_the_union.txt') as f:
    state_of_the_union = f.read()

from langchain.text_splitter import SpacyTextSplitter
text_splitter = SpacyTextSplitter(chunk_size=1000)

texts = text_splitter.split_text(state_of_the_union)
print(texts[0])

    Madam Speaker, Madam Vice President, our First Lady and Second Gentleman.
    
    Members of Congress and the Cabinet.
    
    Justices of the Supreme Court.
    
    My fellow Americans.  
    
    
    
    Last year COVID-19 kept us apart.
    
    This year we are finally together again. 
    
    
    
    Tonight, we meet as Democrats Republicans and Independents.
    
    But most importantly as Americans. 
    
    
    
    With a duty to one another to the American people to the Constitution. 
    
    
    
    And with an unwavering resolve that freedom will always triumph over tyranny. 
    
    
    
    Six days ago, Russia’s Vladimir Putin sought to shake the foundations of the free world thinking he could make it bend to his menacing ways.
    
    But he badly miscalculated. 
    
    
    
    He thought he could roll into Ukraine and the world would roll over.
    
    Instead he met a wall of strength he never imagined. 
    
    
    
    He met the Ukrainian people. 
    
    
    
    From President Zelenskyy to every Ukrainian, their fearlessness, their courage, their determination, inspires the world.

# spaCy

spaCy