Compare commits
2 Commits
version3.9
...
mini-core
| Author | SHA1 | Date | |
|---|---|---|---|
|
8a01c56dd1 | ||
|
|
16567d3a44 |
56
.github/workflows/conda-pack-windows.yml
vendored
56
.github/workflows/conda-pack-windows.yml
vendored
@@ -1,56 +0,0 @@
|
|||||||
name: Create Conda Environment Package
|
|
||||||
|
|
||||||
on:
|
|
||||||
workflow_dispatch:
|
|
||||||
|
|
||||||
jobs:
|
|
||||||
build:
|
|
||||||
runs-on: windows-latest
|
|
||||||
|
|
||||||
steps:
|
|
||||||
- name: Checkout repository
|
|
||||||
uses: actions/checkout@v4
|
|
||||||
|
|
||||||
- name: Setup Miniconda
|
|
||||||
uses: conda-incubator/setup-miniconda@v3
|
|
||||||
with:
|
|
||||||
auto-activate-base: true
|
|
||||||
activate-environment: ""
|
|
||||||
|
|
||||||
- name: Create new Conda environment
|
|
||||||
shell: bash -l {0}
|
|
||||||
run: |
|
|
||||||
conda create -n gpt python=3.11 -y
|
|
||||||
conda activate gpt
|
|
||||||
|
|
||||||
- name: Install requirements
|
|
||||||
shell: bash -l {0}
|
|
||||||
run: |
|
|
||||||
conda activate gpt
|
|
||||||
pip install -r requirements.txt
|
|
||||||
|
|
||||||
- name: Install conda-pack
|
|
||||||
shell: bash -l {0}
|
|
||||||
run: |
|
|
||||||
conda activate gpt
|
|
||||||
conda install conda-pack -y
|
|
||||||
|
|
||||||
- name: Pack conda environment
|
|
||||||
shell: bash -l {0}
|
|
||||||
run: |
|
|
||||||
conda activate gpt
|
|
||||||
conda pack -n gpt -o gpt.tar.gz
|
|
||||||
|
|
||||||
- name: Create workspace zip
|
|
||||||
shell: pwsh
|
|
||||||
run: |
|
|
||||||
mkdir workspace
|
|
||||||
Get-ChildItem -Exclude "workspace" | Copy-Item -Destination workspace -Recurse
|
|
||||||
Remove-Item -Path workspace/.git* -Recurse -Force -ErrorAction SilentlyContinue
|
|
||||||
Copy-Item gpt.tar.gz workspace/ -Force
|
|
||||||
|
|
||||||
- name: Upload packed files
|
|
||||||
uses: actions/upload-artifact@v4
|
|
||||||
with:
|
|
||||||
name: gpt-academic-package
|
|
||||||
path: workspace
|
|
||||||
457
README.md
457
README.md
@@ -1,457 +0,0 @@
|
|||||||
> [!IMPORTANT]
|
|
||||||
> `frontier开发分支`最新动态(2024.12.9): 更新对话时间线功能，优化xelatex论文翻译
|
|
||||||
> `wiki文档`最新动态(2024.12.5): 更新ollama接入指南
|
|
||||||
>
|
|
||||||
> 2024.10.10: 突发停电，紧急恢复了提供[whl包](https://drive.google.com/file/d/19U_hsLoMrjOlQSzYS3pzWX9fTzyusArP/view?usp=sharing)的文件服务器
|
|
||||||
> 2024.10.8: 版本3.90加入对llama-index的初步支持，版本3.80加入插件二级菜单功能（详见wiki）
|
|
||||||
> 2024.5.1: 加入Doc2x翻译PDF论文的功能，[查看详情](https://github.com/binary-husky/gpt_academic/wiki/Doc2x)
|
|
||||||
> 2024.3.11: 全力支持Qwen、GLM、DeepseekCoder等中文大语言模型！ SoVits语音克隆模块，[查看详情](https://www.bilibili.com/video/BV1Rp421S7tF/)
|
|
||||||
> 2024.1.17: 安装依赖时，请选择`requirements.txt`中**指定的版本**。 安装命令：`pip install -r requirements.txt`。本项目完全开源免费，您可通过订阅[在线服务](https://github.com/binary-husky/gpt_academic/wiki/online)的方式鼓励本项目的发展。
|
|
||||||
|
|
||||||
<br>
|
|
||||||
|
|
||||||
<div align=center>
|
|
||||||
<h1 aligh="center">
|
|
||||||
<img src="docs/logo.png" width="40"> GPT 学术优化 (GPT Academic)
|
|
||||||
</h1>
|
|
||||||
|
|
||||||
[![Github][Github-image]][Github-url]
|
|
||||||
[![License][License-image]][License-url]
|
|
||||||
[![Releases][Releases-image]][Releases-url]
|
|
||||||
[![Installation][Installation-image]][Installation-url]
|
|
||||||
[![Wiki][Wiki-image]][Wiki-url]
|
|
||||||
[![PR][PRs-image]][PRs-url]
|
|
||||||
|
|
||||||
[Github-image]: https://img.shields.io/badge/github-12100E.svg?style=flat-square
|
|
||||||
[License-image]: https://img.shields.io/github/license/binary-husky/gpt_academic?label=License&style=flat-square&color=orange
|
|
||||||
[Releases-image]: https://img.shields.io/github/release/binary-husky/gpt_academic?label=Release&style=flat-square&color=blue
|
|
||||||
[Installation-image]: https://img.shields.io/badge/dynamic/json?color=blue&url=https://raw.githubusercontent.com/binary-husky/gpt_academic/master/version&query=$.version&label=Installation&style=flat-square
|
|
||||||
[Wiki-image]: https://img.shields.io/badge/wiki-项目文档-black?style=flat-square
|
|
||||||
[PRs-image]: https://img.shields.io/badge/PRs-welcome-pink?style=flat-square
|
|
||||||
|
|
||||||
[Github-url]: https://github.com/binary-husky/gpt_academic
|
|
||||||
[License-url]: https://github.com/binary-husky/gpt_academic/blob/master/LICENSE
|
|
||||||
[Releases-url]: https://github.com/binary-husky/gpt_academic/releases
|
|
||||||
[Installation-url]: https://github.com/binary-husky/gpt_academic#installation
|
|
||||||
[Wiki-url]: https://github.com/binary-husky/gpt_academic/wiki
|
|
||||||
[PRs-url]: https://github.com/binary-husky/gpt_academic/pulls
|
|
||||||
|
|
||||||
|
|
||||||
</div>
|
|
||||||
<br>
|
|
||||||
|
|
||||||
**如果喜欢这个项目，请给它一个Star；如果您发明了好用的快捷键或插件，欢迎发pull requests！**
|
|
||||||
|
|
||||||
If you like this project, please give it a Star.
|
|
||||||
Read this in [English](docs/README.English.md) | [日本語](docs/README.Japanese.md) | [한국어](docs/README.Korean.md) | [Русский](docs/README.Russian.md) | [Français](docs/README.French.md). All translations have been provided by the project itself. To translate this project to arbitrary language with GPT, read and run [`multi_language.py`](multi_language.py) (experimental).
|
|
||||||
<br>
|
|
||||||
|
|
||||||
> [!NOTE]
|
|
||||||
> 1.本项目中每个文件的功能都在[自译解报告](https://github.com/binary-husky/gpt_academic/wiki/GPT‐Academic项目自译解报告)`self_analysis.md`详细说明。随着版本的迭代，您也可以随时自行点击相关函数插件，调用GPT重新生成项目的自我解析报告。常见问题请查阅wiki。
|
|
||||||
> [](#installation) [](https://github.com/binary-husky/gpt_academic/releases) [](https://github.com/binary-husky/gpt_academic/wiki/项目配置说明) []([https://github.com/binary-husky/gpt_academic/wiki/项目配置说明](https://github.com/binary-husky/gpt_academic/wiki))
|
|
||||||
>
|
|
||||||
> 2.本项目兼容并鼓励尝试国内中文大语言基座模型如通义千问，智谱GLM等。支持多个api-key共存，可在配置文件中填写如`API_KEY="openai-key1,openai-key2,azure-key3,api2d-key4"`。需要临时更换`API_KEY`时，在输入区输入临时的`API_KEY`然后回车键提交即可生效。
|
|
||||||
|
|
||||||
<br><br>
|
|
||||||
|
|
||||||
<div align="center">
|
|
||||||
|
|
||||||
功能（⭐= 近期新增功能） | 描述
|
|
||||||
--- | ---
|
|
||||||
⭐[接入新模型](https://github.com/binary-husky/gpt_academic/wiki/%E5%A6%82%E4%BD%95%E5%88%87%E6%8D%A2%E6%A8%A1%E5%9E%8B) | 百度[千帆](https://cloud.baidu.com/doc/WENXINWORKSHOP/s/Nlks5zkzu)与文心一言, 通义千问[Qwen](https://modelscope.cn/models/qwen/Qwen-7B-Chat/summary)，上海AI-Lab[书生](https://github.com/InternLM/InternLM)，讯飞[星火](https://xinghuo.xfyun.cn/)，[LLaMa2](https://huggingface.co/meta-llama/Llama-2-7b-chat-hf)，[智谱GLM4](https://open.bigmodel.cn/)，DALLE3, [DeepseekCoder](https://coder.deepseek.com/)
|
|
||||||
⭐支持mermaid图像渲染 | 支持让GPT生成[流程图](https://www.bilibili.com/video/BV18c41147H9/)、状态转移图、甘特图、饼状图、GitGraph等等（3.7版本）
|
|
||||||
⭐Arxiv论文精细翻译 ([Docker](https://github.com/binary-husky/gpt_academic/pkgs/container/gpt_academic_with_latex)) | [插件] 一键[以超高质量翻译arxiv论文](https://www.bilibili.com/video/BV1dz4y1v77A/)，目前最好的论文翻译工具
|
|
||||||
⭐[实时语音对话输入](https://github.com/binary-husky/gpt_academic/blob/master/docs/use_audio.md) | [插件] 异步[监听音频](https://www.bilibili.com/video/BV1AV4y187Uy/)，自动断句，自动寻找回答时机
|
|
||||||
⭐AutoGen多智能体插件 | [插件] 借助微软AutoGen，探索多Agent的智能涌现可能！
|
|
||||||
⭐虚空终端插件 | [插件] 能够使用自然语言直接调度本项目其他插件
|
|
||||||
润色、翻译、代码解释 | 一键润色、翻译、查找论文语法错误、解释代码
|
|
||||||
[自定义快捷键](https://www.bilibili.com/video/BV14s4y1E7jN) | 支持自定义快捷键
|
|
||||||
模块化设计 | 支持自定义强大的[插件](https://github.com/binary-husky/gpt_academic/tree/master/crazy_functions)，插件支持[热更新](https://github.com/binary-husky/gpt_academic/wiki/%E5%87%BD%E6%95%B0%E6%8F%92%E4%BB%B6%E6%8C%87%E5%8D%97)
|
|
||||||
[程序剖析](https://www.bilibili.com/video/BV1cj411A7VW) | [插件] 一键剖析Python/C/C++/Java/Lua/...项目树 或 [自我剖析](https://www.bilibili.com/video/BV1cj411A7VW)
|
|
||||||
读论文、[翻译](https://www.bilibili.com/video/BV1KT411x7Wn)论文 | [插件] 一键解读latex/pdf论文全文并生成摘要
|
|
||||||
Latex全文[翻译](https://www.bilibili.com/video/BV1nk4y1Y7Js/)、[润色](https://www.bilibili.com/video/BV1FT411H7c5/) | [插件] 一键翻译或润色latex论文
|
|
||||||
批量注释生成 | [插件] 一键批量生成函数注释
|
|
||||||
Markdown[中英互译](https://www.bilibili.com/video/BV1yo4y157jV/) | [插件] 看到上面5种语言的[README](https://github.com/binary-husky/gpt_academic/blob/master/docs/README.English.md)了吗？就是出自他的手笔
|
|
||||||
[PDF论文全文翻译功能](https://www.bilibili.com/video/BV1KT411x7Wn) | [插件] PDF论文提取题目&摘要+翻译全文（多线程）
|
|
||||||
[Arxiv小助手](https://www.bilibili.com/video/BV1LM4y1279X) | [插件] 输入arxiv文章url即可一键翻译摘要+下载PDF
|
|
||||||
Latex论文一键校对 | [插件] 仿Grammarly对Latex文章进行语法、拼写纠错+输出对照PDF
|
|
||||||
[谷歌学术统合小助手](https://www.bilibili.com/video/BV19L411U7ia) | [插件] 给定任意谷歌学术搜索页面URL，让gpt帮你[写relatedworks](https://www.bilibili.com/video/BV1GP411U7Az/)
|
|
||||||
互联网信息聚合+GPT | [插件] 一键[让GPT从互联网获取信息](https://www.bilibili.com/video/BV1om4y127ck)回答问题，让信息永不过时
|
|
||||||
公式/图片/表格显示 | 可以同时显示公式的[tex形式和渲染形式](https://user-images.githubusercontent.com/96192199/230598842-1d7fcddd-815d-40ee-af60-baf488a199df.png)，支持公式、代码高亮
|
|
||||||
启动暗色[主题](https://github.com/binary-husky/gpt_academic/issues/173) | 在浏览器url后面添加```/?__theme=dark```可以切换dark主题
|
|
||||||
[多LLM模型](https://www.bilibili.com/video/BV1wT411p7yf)支持 | 同时被GPT3.5、GPT4、[清华ChatGLM2](https://github.com/THUDM/ChatGLM2-6B)、[复旦MOSS](https://github.com/OpenLMLab/MOSS)伺候的感觉一定会很不错吧？
|
|
||||||
更多LLM模型接入，支持[huggingface部署](https://huggingface.co/spaces/qingxu98/gpt-academic) | 加入Newbing接口(新必应)，引入清华[Jittorllms](https://github.com/Jittor/JittorLLMs)支持[LLaMA](https://github.com/facebookresearch/llama)和[盘古α](https://openi.org.cn/pangu/)
|
|
||||||
⭐[void-terminal](https://github.com/binary-husky/void-terminal) pip包 | 脱离GUI，在Python中直接调用本项目的所有函数插件（开发中）
|
|
||||||
更多新功能展示 (图像生成等) …… | 见本文档结尾处 ……
|
|
||||||
</div>
|
|
||||||
|
|
||||||
|
|
||||||
- 新界面（修改`config.py`中的LAYOUT选项即可实现“左右布局”和“上下布局”的切换）
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/279702205-d81137c3-affd-4cd1-bb5e-b15610389762.gif" width="700" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://github.com/binary-husky/gpt_academic/assets/96192199/70ff1ec5-e589-4561-a29e-b831079b37fb.gif" width="700" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
|
|
||||||
- 所有按钮都通过读取functional.py动态生成，可随意加自定义功能，解放剪贴板
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/231975334-b4788e91-4887-412f-8b43-2b9c5f41d248.gif" width="700" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
- 润色/纠错
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/231980294-f374bdcb-3309-4560-b424-38ef39f04ebd.gif" width="700" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
- 如果输出包含公式，会以tex形式和渲染形式同时显示，方便复制和阅读
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/230598842-1d7fcddd-815d-40ee-af60-baf488a199df.png" width="700" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
- 懒得看项目代码？直接把整个工程炫ChatGPT嘴里
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/226935232-6b6a73ce-8900-4aee-93f9-733c7e6fef53.png" width="700" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
- 多种大语言模型混合调用（ChatGLM + OpenAI-GPT3.5 + GPT4）
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/232537274-deca0563-7aa6-4b5d-94a2-b7c453c47794.png" width="700" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
<br><br>
|
|
||||||
|
|
||||||
# Installation
|
|
||||||
|
|
||||||
```mermaid
|
|
||||||
flowchart TD
|
|
||||||
A{"安装方法"} --> W1("I. 🔑直接运行 (Windows, Linux or MacOS)")
|
|
||||||
W1 --> W11["1. Python pip包管理依赖"]
|
|
||||||
W1 --> W12["2. Anaconda包管理依赖（推荐⭐）"]
|
|
||||||
|
|
||||||
A --> W2["II. 🐳使用Docker (Windows, Linux or MacOS)"]
|
|
||||||
|
|
||||||
W2 --> k1["1. 部署项目全部能力的大镜像（推荐⭐）"]
|
|
||||||
W2 --> k2["2. 仅在线模型（GPT, GLM4等）镜像"]
|
|
||||||
W2 --> k3["3. 在线模型 + Latex的大镜像"]
|
|
||||||
|
|
||||||
A --> W4["IV. 🚀其他部署方法"]
|
|
||||||
W4 --> C1["1. Windows/MacOS 一键安装运行脚本（推荐⭐）"]
|
|
||||||
W4 --> C2["2. Huggingface, Sealos远程部署"]
|
|
||||||
W4 --> C4["3. ... 其他 ..."]
|
|
||||||
```
|
|
||||||
|
|
||||||
### 安装方法I：直接运行 (Windows, Linux or MacOS)
|
|
||||||
|
|
||||||
1. 下载项目
|
|
||||||
|
|
||||||
```sh
|
|
||||||
git clone --depth=1 https://github.com/binary-husky/gpt_academic.git
|
|
||||||
cd gpt_academic
|
|
||||||
```
|
|
||||||
|
|
||||||
2. 配置API_KEY等变量
|
|
||||||
|
|
||||||
在`config.py`中，配置API KEY等变量。[特殊网络环境设置方法](https://github.com/binary-husky/gpt_academic/issues/1)、[Wiki-项目配置说明](https://github.com/binary-husky/gpt_academic/wiki/项目配置说明)。
|
|
||||||
|
|
||||||
「 程序会优先检查是否存在名为`config_private.py`的私密配置文件，并用其中的配置覆盖`config.py`的同名配置。如您能理解以上读取逻辑，我们强烈建议您在`config.py`同路径下创建一个名为`config_private.py`的新配置文件，并使用`config_private.py`配置项目，从而确保自动更新时不会丢失配置 」。
|
|
||||||
|
|
||||||
「 支持通过`环境变量`配置项目，环境变量的书写格式参考`docker-compose.yml`文件或者我们的[Wiki页面](https://github.com/binary-husky/gpt_academic/wiki/项目配置说明)。配置读取优先级: `环境变量` > `config_private.py` > `config.py` 」。
|
|
||||||
|
|
||||||
|
|
||||||
3. 安装依赖
|
|
||||||
```sh
|
|
||||||
# （选择I: 如熟悉python, python推荐版本 3.9 ~ 3.11）备注：使用官方pip源或者阿里pip源, 临时换源方法：python -m pip install -r requirements.txt -i https://mirrors.aliyun.com/pypi/simple/
|
|
||||||
python -m pip install -r requirements.txt
|
|
||||||
|
|
||||||
# （选择II: 使用Anaconda）步骤也是类似的 (https://www.bilibili.com/video/BV1rc411W7Dr)：
|
|
||||||
conda create -n gptac_venv python=3.11 # 创建anaconda环境
|
|
||||||
conda activate gptac_venv # 激活anaconda环境
|
|
||||||
python -m pip install -r requirements.txt # 这个步骤和pip安装一样的步骤
|
|
||||||
```
|
|
||||||
|
|
||||||
|
|
||||||
<details><summary>如果需要支持清华ChatGLM系列/复旦MOSS/RWKV作为后端，请点击展开此处</summary>
|
|
||||||
<p>
|
|
||||||
|
|
||||||
【可选步骤】如果需要支持清华ChatGLM系列/复旦MOSS作为后端，需要额外安装更多依赖（前提条件：熟悉Python + 用过Pytorch + 电脑配置够强）：
|
|
||||||
|
|
||||||
```sh
|
|
||||||
# 【可选步骤I】支持清华ChatGLM3。清华ChatGLM备注：如果遇到"Call ChatGLM fail 不能正常加载ChatGLM的参数" 错误，参考如下： 1：以上默认安装的为torch+cpu版，使用cuda需要卸载torch重新安装torch+cuda； 2：如因本机配置不够无法加载模型，可以修改request_llm/bridge_chatglm.py中的模型精度, 将 AutoTokenizer.from_pretrained("THUDM/chatglm-6b", trust_remote_code=True) 都修改为 AutoTokenizer.from_pretrained("THUDM/chatglm-6b-int4", trust_remote_code=True)
|
|
||||||
python -m pip install -r request_llms/requirements_chatglm.txt
|
|
||||||
|
|
||||||
# 【可选步骤II】支持清华ChatGLM4 注意：此模型至少需要24G显存
|
|
||||||
python -m pip install -r request_llms/requirements_chatglm4.txt
|
|
||||||
# 可使用modelscope下载ChatGLM4模型
|
|
||||||
# pip install modelscope
|
|
||||||
# modelscope download --model ZhipuAI/glm-4-9b-chat --local_dir ./THUDM/glm-4-9b-chat
|
|
||||||
|
|
||||||
# 【可选步骤III】支持复旦MOSS
|
|
||||||
python -m pip install -r request_llms/requirements_moss.txt
|
|
||||||
git clone --depth=1 https://github.com/OpenLMLab/MOSS.git request_llms/moss # 注意执行此行代码时，必须处于项目根路径
|
|
||||||
|
|
||||||
# 【可选步骤IV】支持RWKV Runner
|
|
||||||
参考wiki：https://github.com/binary-husky/gpt_academic/wiki/%E9%80%82%E9%85%8DRWKV-Runner
|
|
||||||
|
|
||||||
# 【可选步骤V】确保config.py配置文件的AVAIL_LLM_MODELS包含了期望的模型，目前支持的全部模型如下(jittorllms系列目前仅支持docker方案)：
|
|
||||||
AVAIL_LLM_MODELS = ["gpt-3.5-turbo", "api2d-gpt-3.5-turbo", "gpt-4", "api2d-gpt-4", "chatglm", "moss"] # + ["jittorllms_rwkv", "jittorllms_pangualpha", "jittorllms_llama"]
|
|
||||||
|
|
||||||
# 【可选步骤VI】支持本地模型INT8,INT4量化（这里所指的模型本身不是量化版本，目前deepseek-coder支持，后面测试后会加入更多模型量化选择）
|
|
||||||
pip install bitsandbyte
|
|
||||||
# windows用户安装bitsandbytes需要使用下面bitsandbytes-windows-webui
|
|
||||||
python -m pip install bitsandbytes --prefer-binary --extra-index-url=https://jllllll.github.io/bitsandbytes-windows-webui
|
|
||||||
pip install -U git+https://github.com/huggingface/transformers.git
|
|
||||||
pip install -U git+https://github.com/huggingface/accelerate.git
|
|
||||||
pip install peft
|
|
||||||
```
|
|
||||||
|
|
||||||
</p>
|
|
||||||
</details>
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
4. 运行
|
|
||||||
```sh
|
|
||||||
python main.py
|
|
||||||
```
|
|
||||||
|
|
||||||
### 安装方法II：使用Docker
|
|
||||||
|
|
||||||
0. 部署项目的全部能力（这个是包含cuda和latex的大型镜像。但如果您网速慢、硬盘小，则不推荐该方法部署完整项目）
|
|
||||||
[](https://github.com/binary-husky/gpt_academic/actions/workflows/build-with-all-capacity.yml)
|
|
||||||
|
|
||||||
``` sh
|
|
||||||
# 修改docker-compose.yml，保留方案0并删除其他方案。然后运行：
|
|
||||||
docker-compose up
|
|
||||||
```
|
|
||||||
|
|
||||||
1. 仅ChatGPT + GLM4 + 文心一言+spark等在线模型（推荐大多数人选择）
|
|
||||||
[](https://github.com/binary-husky/gpt_academic/actions/workflows/build-without-local-llms.yml)
|
|
||||||
[](https://github.com/binary-husky/gpt_academic/actions/workflows/build-with-latex.yml)
|
|
||||||
[](https://github.com/binary-husky/gpt_academic/actions/workflows/build-with-audio-assistant.yml)
|
|
||||||
|
|
||||||
``` sh
|
|
||||||
# 修改docker-compose.yml，保留方案1并删除其他方案。然后运行：
|
|
||||||
docker-compose up
|
|
||||||
```
|
|
||||||
|
|
||||||
P.S. 如果需要依赖Latex的插件功能，请见Wiki。另外，您也可以直接使用方案4或者方案0获取Latex功能。
|
|
||||||
|
|
||||||
2. ChatGPT + GLM3 + MOSS + LLAMA2 + 通义千问（需要熟悉[Nvidia Docker](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html#installing-on-ubuntu-and-debian)运行时）
|
|
||||||
[](https://github.com/binary-husky/gpt_academic/actions/workflows/build-with-chatglm.yml)
|
|
||||||
|
|
||||||
``` sh
|
|
||||||
# 修改docker-compose.yml，保留方案2并删除其他方案。然后运行：
|
|
||||||
docker-compose up
|
|
||||||
```
|
|
||||||
|
|
||||||
|
|
||||||
### 安装方法III：其他部署方法
|
|
||||||
1. **Windows一键运行脚本**。
|
|
||||||
完全不熟悉python环境的Windows用户可以下载[Release](https://github.com/binary-husky/gpt_academic/releases)中发布的一键运行脚本安装无本地模型的版本。脚本贡献来源：[oobabooga](https://github.com/oobabooga/one-click-installers)。
|
|
||||||
|
|
||||||
2. 使用第三方API、Azure等、文心一言、星火等，见[Wiki页面](https://github.com/binary-husky/gpt_academic/wiki/项目配置说明)
|
|
||||||
|
|
||||||
3. 云服务器远程部署避坑指南。
|
|
||||||
请访问[云服务器远程部署wiki](https://github.com/binary-husky/gpt_academic/wiki/%E4%BA%91%E6%9C%8D%E5%8A%A1%E5%99%A8%E8%BF%9C%E7%A8%8B%E9%83%A8%E7%BD%B2%E6%8C%87%E5%8D%97)
|
|
||||||
|
|
||||||
4. 在其他平台部署&二级网址部署
|
|
||||||
- 使用Sealos[一键部署](https://github.com/binary-husky/gpt_academic/issues/993)。
|
|
||||||
- 使用WSL2（Windows Subsystem for Linux 子系统）。请访问[部署wiki-2](https://github.com/binary-husky/gpt_academic/wiki/%E4%BD%BF%E7%94%A8WSL2%EF%BC%88Windows-Subsystem-for-Linux-%E5%AD%90%E7%B3%BB%E7%BB%9F%EF%BC%89%E9%83%A8%E7%BD%B2)
|
|
||||||
- 如何在二级网址（如`http://localhost/subpath`）下运行。请访问[FastAPI运行说明](docs/WithFastapi.md)
|
|
||||||
|
|
||||||
<br><br>
|
|
||||||
|
|
||||||
# Advanced Usage
|
|
||||||
### I：自定义新的便捷按钮（学术快捷键）
|
|
||||||
|
|
||||||
现在已可以通过UI中的`界面外观`菜单中的`自定义菜单`添加新的便捷按钮。如果需要在代码中定义，请使用任意文本编辑器打开`core_functional.py`，添加如下条目即可：
|
|
||||||
|
|
||||||
```python
|
|
||||||
"超级英译中": {
|
|
||||||
# 前缀，会被加在你的输入之前。例如，用来描述你的要求，例如翻译、解释代码、润色等等
|
|
||||||
"Prefix": "请翻译把下面一段内容成中文，然后用一个markdown表格逐一解释文中出现的专有名词：\n\n",
|
|
||||||
|
|
||||||
# 后缀，会被加在你的输入之后。例如，配合前缀可以把你的输入内容用引号圈起来。
|
|
||||||
"Suffix": "",
|
|
||||||
},
|
|
||||||
```
|
|
||||||
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/226899272-477c2134-ed71-4326-810c-29891fe4a508.png" width="500" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
### II：自定义函数插件
|
|
||||||
编写强大的函数插件来执行任何你想得到的和想不到的任务。
|
|
||||||
本项目的插件编写、调试难度很低，只要您具备一定的python基础知识，就可以仿照我们提供的模板实现自己的插件功能。
|
|
||||||
详情请参考[函数插件指南](https://github.com/binary-husky/gpt_academic/wiki/%E5%87%BD%E6%95%B0%E6%8F%92%E4%BB%B6%E6%8C%87%E5%8D%97)。
|
|
||||||
|
|
||||||
<br><br>
|
|
||||||
|
|
||||||
# Updates
|
|
||||||
### I：动态
|
|
||||||
|
|
||||||
1. 对话保存功能。在函数插件区调用 `保存当前的对话` 即可将当前对话保存为可读+可复原的html文件，
|
|
||||||
另外在函数插件区（下拉菜单）调用 `载入对话历史存档` ，即可还原之前的会话。
|
|
||||||
Tip：不指定文件直接点击 `载入对话历史存档` 可以查看历史html存档缓存。
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/235222390-24a9acc0-680f-49f5-bc81-2f3161f1e049.png" width="500" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
2. ⭐Latex/Arxiv论文翻译功能⭐
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://github.com/binary-husky/gpt_academic/assets/96192199/002a1a75-ace0-4e6a-94e2-ec1406a746f1" height="250" > ===>
|
|
||||||
<img src="https://github.com/binary-husky/gpt_academic/assets/96192199/9fdcc391-f823-464f-9322-f8719677043b" height="250" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
3. 虚空终端（从自然语言输入中，理解用户意图+自动调用其他插件）
|
|
||||||
|
|
||||||
- 步骤一：输入 “ 请调用插件翻译PDF论文，地址为https://openreview.net/pdf?id=rJl0r3R9KX ”
|
|
||||||
- 步骤二：点击“虚空终端”
|
|
||||||
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://github.com/binary-husky/gpt_academic/assets/96192199/66f1b044-e9ff-4eed-9126-5d4f3668f1ed" width="500" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
4. 模块化功能设计，简单的接口却能支持强大的功能
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/229288270-093643c1-0018-487a-81e6-1d7809b6e90f.png" height="400" >
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/227504931-19955f78-45cd-4d1c-adac-e71e50957915.png" height="400" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
5. 译解其他开源项目
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/226935232-6b6a73ce-8900-4aee-93f9-733c7e6fef53.png" height="250" >
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/226969067-968a27c1-1b9c-486b-8b81-ab2de8d3f88a.png" height="250" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
6. 装饰[live2d](https://github.com/fghrsh/live2d_demo)的小功能（默认关闭，需要修改`config.py`）
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/236432361-67739153-73e8-43fe-8111-b61296edabd9.png" width="500" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
7. OpenAI图像生成
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://github.com/binary-husky/gpt_academic/assets/96192199/bc7ab234-ad90-48a0-8d62-f703d9e74665" width="500" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
8. 基于mermaid的流图、脑图绘制
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://github.com/binary-husky/gpt_academic/assets/96192199/c518b82f-bd53-46e2-baf5-ad1b081c1da4" width="500" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
9. Latex全文校对纠错
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://github.com/binary-husky/gpt_academic/assets/96192199/651ccd98-02c9-4464-91e1-77a6b7d1b033" height="200" > ===>
|
|
||||||
<img src="https://github.com/binary-husky/gpt_academic/assets/96192199/476f66d9-7716-4537-b5c1-735372c25adb" height="200">
|
|
||||||
</div>
|
|
||||||
|
|
||||||
10. 语言、主题切换
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://github.com/binary-husky/gpt_academic/assets/96192199/b6799499-b6fb-4f0c-9c8e-1b441872f4e8" width="500" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
### II：版本:
|
|
||||||
- version 3.80(TODO): 优化AutoGen插件主题并设计一系列衍生插件
|
|
||||||
- version 3.70: 引入Mermaid绘图，实现GPT画脑图等功能
|
|
||||||
- version 3.60: 引入AutoGen作为新一代插件的基石
|
|
||||||
- version 3.57: 支持GLM3，星火v3，文心一言v4，修复本地模型的并发BUG
|
|
||||||
- version 3.56: 支持动态追加基础功能按钮，新汇报PDF汇总页面
|
|
||||||
- version 3.55: 重构前端界面，引入悬浮窗口与菜单栏
|
|
||||||
- version 3.54: 新增动态代码解释器（Code Interpreter）（待完善）
|
|
||||||
- version 3.53: 支持动态选择不同界面主题，提高稳定性&解决多用户冲突问题
|
|
||||||
- version 3.50: 使用自然语言调用本项目的所有函数插件（虚空终端），支持插件分类，改进UI，设计新主题
|
|
||||||
- version 3.49: 支持百度千帆平台和文心一言
|
|
||||||
- version 3.48: 支持阿里达摩院通义千问，上海AI-Lab书生，讯飞星火
|
|
||||||
- version 3.46: 支持完全脱手操作的实时语音对话
|
|
||||||
- version 3.45: 支持自定义ChatGLM2微调模型
|
|
||||||
- version 3.44: 正式支持Azure，优化界面易用性
|
|
||||||
- version 3.4: +arxiv论文翻译、latex论文批改功能
|
|
||||||
- version 3.3: +互联网信息综合功能
|
|
||||||
- version 3.2: 函数插件支持更多参数接口 (保存对话功能, 解读任意语言代码+同时询问任意的LLM组合)
|
|
||||||
- version 3.1: 支持同时问询多个gpt模型！支持api2d，支持多个apikey负载均衡
|
|
||||||
- version 3.0: 对chatglm和其他小型llm的支持
|
|
||||||
- version 2.6: 重构了插件结构，提高了交互性，加入更多插件
|
|
||||||
- version 2.5: 自更新，解决总结大工程源代码时文本过长、token溢出的问题
|
|
||||||
- version 2.4: 新增PDF全文翻译功能; 新增输入区切换位置的功能
|
|
||||||
- version 2.3: 增强多线程交互性
|
|
||||||
- version 2.2: 函数插件支持热重载
|
|
||||||
- version 2.1: 可折叠式布局
|
|
||||||
- version 2.0: 引入模块化函数插件
|
|
||||||
- version 1.0: 基础功能
|
|
||||||
|
|
||||||
GPT Academic开发者QQ群：`610599535`
|
|
||||||
|
|
||||||
- 已知问题
|
|
||||||
- 某些浏览器翻译插件干扰此软件前端的运行
|
|
||||||
- 官方Gradio目前有很多兼容性问题，请**务必使用`requirement.txt`安装Gradio**
|
|
||||||
|
|
||||||
```mermaid
|
|
||||||
timeline LR
|
|
||||||
title GPT-Academic项目发展历程
|
|
||||||
section 2.x
|
|
||||||
1.0~2.2: 基础功能: 引入模块化函数插件: 可折叠式布局: 函数插件支持热重载
|
|
||||||
2.3~2.5: 增强多线程交互性: 新增PDF全文翻译功能: 新增输入区切换位置的功能: 自更新
|
|
||||||
2.6: 重构了插件结构: 提高了交互性: 加入更多插件
|
|
||||||
section 3.x
|
|
||||||
3.0~3.1: 对chatglm支持: 对其他小型llm支持: 支持同时问询多个gpt模型: 支持多个apikey负载均衡
|
|
||||||
3.2~3.3: 函数插件支持更多参数接口: 保存对话功能: 解读任意语言代码: 同时询问任意的LLM组合: 互联网信息综合功能
|
|
||||||
3.4: 加入arxiv论文翻译: 加入latex论文批改功能
|
|
||||||
3.44: 正式支持Azure: 优化界面易用性
|
|
||||||
3.46: 自定义ChatGLM2微调模型: 实时语音对话
|
|
||||||
3.49: 支持阿里达摩院通义千问: 上海AI-Lab书生: 讯飞星火: 支持百度千帆平台 & 文心一言
|
|
||||||
3.50: 虚空终端: 支持插件分类: 改进UI: 设计新主题
|
|
||||||
3.53: 动态选择不同界面主题: 提高稳定性: 解决多用户冲突问题
|
|
||||||
3.55: 动态代码解释器: 重构前端界面: 引入悬浮窗口与菜单栏
|
|
||||||
3.56: 动态追加基础功能按钮: 新汇报PDF汇总页面
|
|
||||||
3.57: GLM3, 星火v3: 支持文心一言v4: 修复本地模型的并发BUG
|
|
||||||
3.60: 引入AutoGen
|
|
||||||
3.70: 引入Mermaid绘图: 实现GPT画脑图等功能
|
|
||||||
3.80(TODO): 优化AutoGen插件主题: 设计衍生插件
|
|
||||||
|
|
||||||
```
|
|
||||||
|
|
||||||
|
|
||||||
### III：主题
|
|
||||||
可以通过修改`THEME`选项（config.py）变更主题
|
|
||||||
1. `Chuanhu-Small-and-Beautiful` [网址](https://github.com/GaiZhenbiao/ChuanhuChatGPT/)
|
|
||||||
|
|
||||||
|
|
||||||
### IV：本项目的开发分支
|
|
||||||
|
|
||||||
1. `master` 分支: 主分支，稳定版
|
|
||||||
2. `frontier` 分支: 开发分支，测试版
|
|
||||||
3. 如何[接入其他大模型](request_llms/README.md)
|
|
||||||
4. 访问GPT-Academic的[在线服务并支持我们](https://github.com/binary-husky/gpt_academic/wiki/online)
|
|
||||||
|
|
||||||
### V：参考与学习
|
|
||||||
|
|
||||||
```
|
|
||||||
代码中参考了很多其他优秀项目中的设计，顺序不分先后：
|
|
||||||
|
|
||||||
# 清华ChatGLM2-6B:
|
|
||||||
https://github.com/THUDM/ChatGLM2-6B
|
|
||||||
|
|
||||||
# 清华JittorLLMs:
|
|
||||||
https://github.com/Jittor/JittorLLMs
|
|
||||||
|
|
||||||
# ChatPaper:
|
|
||||||
https://github.com/kaixindelele/ChatPaper
|
|
||||||
|
|
||||||
# Edge-GPT:
|
|
||||||
https://github.com/acheong08/EdgeGPT
|
|
||||||
|
|
||||||
# ChuanhuChatGPT:
|
|
||||||
https://github.com/GaiZhenbiao/ChuanhuChatGPT
|
|
||||||
|
|
||||||
# Oobabooga one-click installer:
|
|
||||||
https://github.com/oobabooga/one-click-installers
|
|
||||||
|
|
||||||
# More：
|
|
||||||
https://github.com/gradio-app/gradio
|
|
||||||
https://github.com/fghrsh/live2d_demo
|
|
||||||
```
|
|
||||||
18
config.py
18
config.py
@@ -36,7 +36,7 @@ AVAIL_LLM_MODELS = ["gpt-4-1106-preview", "gpt-4-turbo-preview", "gpt-4-vision-p
|
|||||||
"gpt-4o", "gpt-4o-mini", "gpt-4-turbo", "gpt-4-turbo-2024-04-09",
|
"gpt-4o", "gpt-4o-mini", "gpt-4-turbo", "gpt-4-turbo-2024-04-09",
|
||||||
"gpt-3.5-turbo-1106", "gpt-3.5-turbo-16k", "gpt-3.5-turbo", "azure-gpt-3.5",
|
"gpt-3.5-turbo-1106", "gpt-3.5-turbo-16k", "gpt-3.5-turbo", "azure-gpt-3.5",
|
||||||
"gpt-4", "gpt-4-32k", "azure-gpt-4", "glm-4", "glm-4v", "glm-3-turbo",
|
"gpt-4", "gpt-4-32k", "azure-gpt-4", "glm-4", "glm-4v", "glm-3-turbo",
|
||||||
"gemini-1.5-pro", "chatglm3", "chatglm4"
|
"gemini-1.5-pro", "chatglm3"
|
||||||
]
|
]
|
||||||
|
|
||||||
EMBEDDING_MODEL = "text-embedding-3-small"
|
EMBEDDING_MODEL = "text-embedding-3-small"
|
||||||
@@ -55,7 +55,6 @@ EMBEDDING_MODEL = "text-embedding-3-small"
|
|||||||
# "deepseek-chat" ,"deepseek-coder",
|
# "deepseek-chat" ,"deepseek-coder",
|
||||||
# "gemini-1.5-flash",
|
# "gemini-1.5-flash",
|
||||||
# "yi-34b-chat-0205","yi-34b-chat-200k","yi-large","yi-medium","yi-spark","yi-large-turbo","yi-large-preview",
|
# "yi-34b-chat-0205","yi-34b-chat-200k","yi-large","yi-medium","yi-spark","yi-large-turbo","yi-large-preview",
|
||||||
# "grok-beta",
|
|
||||||
# ]
|
# ]
|
||||||
# --- --- --- ---
|
# --- --- --- ---
|
||||||
# 此外，您还可以在接入one-api/vllm/ollama/Openroute时，
|
# 此外，您还可以在接入one-api/vllm/ollama/Openroute时，
|
||||||
@@ -143,9 +142,6 @@ BAIDU_CLOUD_SECRET_KEY = ''
|
|||||||
BAIDU_CLOUD_QIANFAN_MODEL = 'ERNIE-Bot' # 可选 "ERNIE-Bot-4"(文心大模型4.0), "ERNIE-Bot"(文心一言), "ERNIE-Bot-turbo", "BLOOMZ-7B", "Llama-2-70B-Chat", "Llama-2-13B-Chat", "Llama-2-7B-Chat", "ERNIE-Speed-128K", "ERNIE-Speed-8K", "ERNIE-Lite-8K"
|
BAIDU_CLOUD_QIANFAN_MODEL = 'ERNIE-Bot' # 可选 "ERNIE-Bot-4"(文心大模型4.0), "ERNIE-Bot"(文心一言), "ERNIE-Bot-turbo", "BLOOMZ-7B", "Llama-2-70B-Chat", "Llama-2-13B-Chat", "Llama-2-7B-Chat", "ERNIE-Speed-128K", "ERNIE-Speed-8K", "ERNIE-Lite-8K"
|
||||||
|
|
||||||
|
|
||||||
# 如果使用ChatGLM3或ChatGLM4本地模型，请把 LLM_MODEL="chatglm3" 或LLM_MODEL="chatglm4"，并在此处指定模型路径
|
|
||||||
CHATGLM_LOCAL_MODEL_PATH = "THUDM/glm-4-9b-chat" # 例如"/home/hmp/ChatGLM3-6B/"
|
|
||||||
|
|
||||||
# 如果使用ChatGLM2微调模型，请把 LLM_MODEL="chatglmft"，并在此处指定模型路径
|
# 如果使用ChatGLM2微调模型，请把 LLM_MODEL="chatglmft"，并在此处指定模型路径
|
||||||
CHATGLM_PTUNING_CHECKPOINT = "" # 例如"/home/hmp/ChatGLM2-6B/ptuning/output/6b-pt-128-1e-2/checkpoint-100"
|
CHATGLM_PTUNING_CHECKPOINT = "" # 例如"/home/hmp/ChatGLM2-6B/ptuning/output/6b-pt-128-1e-2/checkpoint-100"
|
||||||
|
|
||||||
@@ -238,6 +234,7 @@ MOONSHOT_API_KEY = ""
|
|||||||
# 零一万物(Yi Model) API KEY
|
# 零一万物(Yi Model) API KEY
|
||||||
YIMODEL_API_KEY = ""
|
YIMODEL_API_KEY = ""
|
||||||
|
|
||||||
|
|
||||||
# 深度求索(DeepSeek) API KEY，默认请求地址为"https://api.deepseek.com/v1/chat/completions"
|
# 深度求索(DeepSeek) API KEY，默认请求地址为"https://api.deepseek.com/v1/chat/completions"
|
||||||
DEEPSEEK_API_KEY = ""
|
DEEPSEEK_API_KEY = ""
|
||||||
|
|
||||||
@@ -245,8 +242,6 @@ DEEPSEEK_API_KEY = ""
|
|||||||
# 紫东太初大模型 https://ai-maas.wair.ac.cn
|
# 紫东太初大模型 https://ai-maas.wair.ac.cn
|
||||||
TAICHU_API_KEY = ""
|
TAICHU_API_KEY = ""
|
||||||
|
|
||||||
# Grok API KEY
|
|
||||||
GROK_API_KEY = ""
|
|
||||||
|
|
||||||
# Mathpix 拥有执行PDF的OCR功能，但是需要注册账号
|
# Mathpix 拥有执行PDF的OCR功能，但是需要注册账号
|
||||||
MATHPIX_APPID = ""
|
MATHPIX_APPID = ""
|
||||||
@@ -278,8 +273,8 @@ GROBID_URLS = [
|
|||||||
]
|
]
|
||||||
|
|
||||||
|
|
||||||
# Searxng互联网检索服务（这是一个huggingface空间，请前往huggingface复制该空间，然后把自己新的空间地址填在这里）
|
# Searxng互联网检索服务
|
||||||
SEARXNG_URLS = [ f"https://kaletianlre-beardvs{i}dd.hf.space/" for i in range(1,5) ]
|
SEARXNG_URL = "https://cloud-1.agent-matrix.com/"
|
||||||
|
|
||||||
|
|
||||||
# 是否允许通过自然语言描述修改本页的配置，该功能具有一定的危险性，默认关闭
|
# 是否允许通过自然语言描述修改本页的配置，该功能具有一定的危险性，默认关闭
|
||||||
@@ -316,7 +311,7 @@ NUM_CUSTOM_BASIC_BTN = 4
|
|||||||
|
|
||||||
|
|
||||||
# 媒体智能体的服务地址（这是一个huggingface空间，请前往huggingface复制该空间，然后把自己新的空间地址填在这里）
|
# 媒体智能体的服务地址（这是一个huggingface空间，请前往huggingface复制该空间，然后把自己新的空间地址填在这里）
|
||||||
DAAS_SERVER_URLS = [ f"https://niuziniu-biligpt{i}.hf.space/stream" for i in range(1,5) ]
|
DAAS_SERVER_URL = "https://hamercity-bbdown.hf.space/stream"
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -378,7 +373,6 @@ DAAS_SERVER_URLS = [ f"https://niuziniu-biligpt{i}.hf.space/stream" for i in ran
|
|||||||
|
|
||||||
本地大模型示意图
|
本地大模型示意图
|
||||||
│
|
│
|
||||||
├── "chatglm4"
|
|
||||||
├── "chatglm3"
|
├── "chatglm3"
|
||||||
├── "chatglm"
|
├── "chatglm"
|
||||||
├── "chatglm_onnx"
|
├── "chatglm_onnx"
|
||||||
@@ -409,7 +403,7 @@ DAAS_SERVER_URLS = [ f"https://niuziniu-biligpt{i}.hf.space/stream" for i in ran
|
|||||||
插件在线服务配置依赖关系示意图
|
插件在线服务配置依赖关系示意图
|
||||||
│
|
│
|
||||||
├── 互联网检索
|
├── 互联网检索
|
||||||
│ └── SEARXNG_URLS
|
│ └── SEARXNG_URL
|
||||||
│
|
│
|
||||||
├── 语音功能
|
├── 语音功能
|
||||||
│ ├── ENABLE_AUDIO
|
│ ├── ENABLE_AUDIO
|
||||||
|
|||||||
@@ -3,744 +3,18 @@ from toolbox import trimmed_format_exc
|
|||||||
from loguru import logger
|
from loguru import logger
|
||||||
|
|
||||||
def get_crazy_functions():
|
def get_crazy_functions():
|
||||||
from crazy_functions.读文章写摘要 import 读文章写摘要
|
from crazy_functions.AntFin import AntFinTest
|
||||||
from crazy_functions.生成函数注释 import 批量生成函数注释
|
|
||||||
from crazy_functions.SourceCode_Analyse import 解析项目本身
|
|
||||||
from crazy_functions.SourceCode_Analyse import 解析一个Python项目
|
|
||||||
from crazy_functions.SourceCode_Analyse import 解析一个Matlab项目
|
|
||||||
from crazy_functions.SourceCode_Analyse import 解析一个C项目的头文件
|
|
||||||
from crazy_functions.SourceCode_Analyse import 解析一个C项目
|
|
||||||
from crazy_functions.SourceCode_Analyse import 解析一个Golang项目
|
|
||||||
from crazy_functions.SourceCode_Analyse import 解析一个Rust项目
|
|
||||||
from crazy_functions.SourceCode_Analyse import 解析一个Java项目
|
|
||||||
from crazy_functions.SourceCode_Analyse import 解析一个前端项目
|
|
||||||
from crazy_functions.高级功能函数模板 import 高阶功能模板函数
|
|
||||||
from crazy_functions.高级功能函数模板 import Demo_Wrap
|
|
||||||
from crazy_functions.Latex_Project_Polish import Latex英文润色
|
|
||||||
from crazy_functions.询问多个大语言模型 import 同时问询
|
|
||||||
from crazy_functions.SourceCode_Analyse import 解析一个Lua项目
|
|
||||||
from crazy_functions.SourceCode_Analyse import 解析一个CSharp项目
|
|
||||||
from crazy_functions.总结word文档 import 总结word文档
|
|
||||||
from crazy_functions.解析JupyterNotebook import 解析ipynb文件
|
|
||||||
from crazy_functions.Conversation_To_File import 载入对话历史存档
|
|
||||||
from crazy_functions.Conversation_To_File import 对话历史存档
|
|
||||||
from crazy_functions.Conversation_To_File import Conversation_To_File_Wrap
|
|
||||||
from crazy_functions.Conversation_To_File import 删除所有本地对话历史记录
|
|
||||||
from crazy_functions.辅助功能 import 清除缓存
|
|
||||||
from crazy_functions.Markdown_Translate import Markdown英译中
|
|
||||||
from crazy_functions.批量总结PDF文档 import 批量总结PDF文档
|
|
||||||
from crazy_functions.PDF_Translate import 批量翻译PDF文档
|
|
||||||
from crazy_functions.谷歌检索小助手 import 谷歌检索小助手
|
|
||||||
from crazy_functions.理解PDF文档内容 import 理解PDF文档内容标准文件输入
|
|
||||||
from crazy_functions.Latex_Project_Polish import Latex中文润色
|
|
||||||
from crazy_functions.Latex_Project_Polish import Latex英文纠错
|
|
||||||
from crazy_functions.Markdown_Translate import Markdown中译英
|
|
||||||
from crazy_functions.虚空终端 import 虚空终端
|
|
||||||
from crazy_functions.生成多种Mermaid图表 import Mermaid_Gen
|
|
||||||
from crazy_functions.PDF_Translate_Wrap import PDF_Tran
|
|
||||||
from crazy_functions.Latex_Function import Latex英文纠错加PDF对比
|
|
||||||
from crazy_functions.Latex_Function import Latex翻译中文并重新编译PDF
|
|
||||||
from crazy_functions.Latex_Function import PDF翻译中文并重新编译PDF
|
|
||||||
from crazy_functions.Latex_Function_Wrap import Arxiv_Localize
|
|
||||||
from crazy_functions.Latex_Function_Wrap import PDF_Localize
|
|
||||||
from crazy_functions.Internet_GPT import 连接网络回答问题
|
|
||||||
from crazy_functions.Internet_GPT_Wrap import NetworkGPT_Wrap
|
|
||||||
from crazy_functions.Image_Generate import 图片生成_DALLE2, 图片生成_DALLE3, 图片修改_DALLE2
|
|
||||||
from crazy_functions.Image_Generate_Wrap import ImageGen_Wrap
|
|
||||||
from crazy_functions.SourceCode_Comment import 注释Python项目
|
|
||||||
from crazy_functions.SourceCode_Comment_Wrap import SourceCodeComment_Wrap
|
|
||||||
from crazy_functions.VideoResource_GPT import 多媒体任务
|
|
||||||
|
|
||||||
function_plugins = {
|
function_plugins = {
|
||||||
"多媒体智能体": {
|
"蚂小财测试": {
|
||||||
"Group": "智能体",
|
"Group": "智能体",
|
||||||
"Color": "stop",
|
"Color": "stop",
|
||||||
"AsButton": False,
|
"AsButton": False,
|
||||||
"Info": "【仅测试】多媒体任务",
|
"Info": "蚂小财测试",
|
||||||
"Function": HotReload(多媒体任务),
|
"Function": HotReload(AntFinTest),
|
||||||
},
|
},
|
||||||
"虚空终端": {
|
|
||||||
"Group": "对话|编程|学术|智能体",
|
|
||||||
"Color": "stop",
|
|
||||||
"AsButton": True,
|
|
||||||
"Info": "使用自然语言实现您的想法",
|
|
||||||
"Function": HotReload(虚空终端),
|
|
||||||
},
|
|
||||||
"解析整个Python项目": {
|
|
||||||
"Group": "编程",
|
|
||||||
"Color": "stop",
|
|
||||||
"AsButton": True,
|
|
||||||
"Info": "解析一个Python项目的所有源文件(.py) | 输入参数为路径",
|
|
||||||
"Function": HotReload(解析一个Python项目),
|
|
||||||
},
|
|
||||||
"注释Python项目": {
|
|
||||||
"Group": "编程",
|
|
||||||
"Color": "stop",
|
|
||||||
"AsButton": False,
|
|
||||||
"Info": "上传一系列python源文件(或者压缩包), 为这些代码添加docstring | 输入参数为路径",
|
|
||||||
"Function": HotReload(注释Python项目),
|
|
||||||
"Class": SourceCodeComment_Wrap,
|
|
||||||
},
|
|
||||||
"载入对话历史存档（先上传存档或输入路径）": {
|
|
||||||
"Group": "对话",
|
|
||||||
"Color": "stop",
|
|
||||||
"AsButton": False,
|
|
||||||
"Info": "载入对话历史存档 | 输入参数为路径",
|
|
||||||
"Function": HotReload(载入对话历史存档),
|
|
||||||
},
|
|
||||||
"删除所有本地对话历史记录（谨慎操作）": {
|
|
||||||
"Group": "对话",
|
|
||||||
"AsButton": False,
|
|
||||||
"Info": "删除所有本地对话历史记录，谨慎操作 | 不需要输入参数",
|
|
||||||
"Function": HotReload(删除所有本地对话历史记录),
|
|
||||||
},
|
|
||||||
"清除所有缓存文件（谨慎操作）": {
|
|
||||||
"Group": "对话",
|
|
||||||
"Color": "stop",
|
|
||||||
"AsButton": False, # 加入下拉菜单中
|
|
||||||
"Info": "清除所有缓存文件，谨慎操作 | 不需要输入参数",
|
|
||||||
"Function": HotReload(清除缓存),
|
|
||||||
},
|
|
||||||
"生成多种Mermaid图表(从当前对话或路径(.pdf/.md/.docx)中生产图表）": {
|
|
||||||
"Group": "对话",
|
|
||||||
"Color": "stop",
|
|
||||||
"AsButton": False,
|
|
||||||
"Info" : "基于当前对话或文件生成多种Mermaid图表,图表类型由模型判断",
|
|
||||||
"Function": None,
|
|
||||||
"Class": Mermaid_Gen
|
|
||||||
},
|
|
||||||
"Arxiv论文翻译": {
|
|
||||||
"Group": "学术",
|
|
||||||
"Color": "stop",
|
|
||||||
"AsButton": True,
|
|
||||||
"Info": "Arixv论文精细翻译 | 输入参数arxiv论文的ID，比如1812.10695",
|
|
||||||
"Function": HotReload(Latex翻译中文并重新编译PDF), # 当注册Class后，Function旧接口仅会在“虚空终端”中起作用
|
|
||||||
"Class": Arxiv_Localize, # 新一代插件需要注册Class
|
|
||||||
},
|
|
||||||
"批量总结Word文档": {
|
|
||||||
"Group": "学术",
|
|
||||||
"Color": "stop",
|
|
||||||
"AsButton": False,
|
|
||||||
"Info": "批量总结word文档 | 输入参数为路径",
|
|
||||||
"Function": HotReload(总结word文档),
|
|
||||||
},
|
|
||||||
"解析整个Matlab项目": {
|
|
||||||
"Group": "编程",
|
|
||||||
"Color": "stop",
|
|
||||||
"AsButton": False,
|
|
||||||
"Info": "解析一个Matlab项目的所有源文件(.m) | 输入参数为路径",
|
|
||||||
"Function": HotReload(解析一个Matlab项目),
|
|
||||||
},
|
|
||||||
"解析整个C++项目头文件": {
|
|
||||||
"Group": "编程",
|
|
||||||
"Color": "stop",
|
|
||||||
"AsButton": False, # 加入下拉菜单中
|
|
||||||
"Info": "解析一个C++项目的所有头文件(.h/.hpp) | 输入参数为路径",
|
|
||||||
"Function": HotReload(解析一个C项目的头文件),
|
|
||||||
},
|
|
||||||
"解析整个C++项目（.cpp/.hpp/.c/.h）": {
|
|
||||||
"Group": "编程",
|
|
||||||
"Color": "stop",
|
|
||||||
"AsButton": False, # 加入下拉菜单中
|
|
||||||
"Info": "解析一个C++项目的所有源文件（.cpp/.hpp/.c/.h）| 输入参数为路径",
|
|
||||||
"Function": HotReload(解析一个C项目),
|
|
||||||
},
|
|
||||||
"解析整个Go项目": {
|
|
||||||
"Group": "编程",
|
|
||||||
"Color": "stop",
|
|
||||||
"AsButton": False, # 加入下拉菜单中
|
|
||||||
"Info": "解析一个Go项目的所有源文件 | 输入参数为路径",
|
|
||||||
"Function": HotReload(解析一个Golang项目),
|
|
||||||
},
|
|
||||||
"解析整个Rust项目": {
|
|
||||||
"Group": "编程",
|
|
||||||
"Color": "stop",
|
|
||||||
"AsButton": False, # 加入下拉菜单中
|
|
||||||
"Info": "解析一个Rust项目的所有源文件 | 输入参数为路径",
|
|
||||||
"Function": HotReload(解析一个Rust项目),
|
|
||||||
},
|
|
||||||
"解析整个Java项目": {
|
|
||||||
"Group": "编程",
|
|
||||||
"Color": "stop",
|
|
||||||
"AsButton": False, # 加入下拉菜单中
|
|
||||||
"Info": "解析一个Java项目的所有源文件 | 输入参数为路径",
|
|
||||||
"Function": HotReload(解析一个Java项目),
|
|
||||||
},
|
|
||||||
"解析整个前端项目（js,ts,css等）": {
|
|
||||||
"Group": "编程",
|
|
||||||
"Color": "stop",
|
|
||||||
"AsButton": False, # 加入下拉菜单中
|
|
||||||
"Info": "解析一个前端项目的所有源文件（js,ts,css等） | 输入参数为路径",
|
|
||||||
"Function": HotReload(解析一个前端项目),
|
|
||||||
},
|
|
||||||
"解析整个Lua项目": {
|
|
||||||
"Group": "编程",
|
|
||||||
"Color": "stop",
|
|
||||||
"AsButton": False, # 加入下拉菜单中
|
|
||||||
"Info": "解析一个Lua项目的所有源文件 | 输入参数为路径",
|
|
||||||
"Function": HotReload(解析一个Lua项目),
|
|
||||||
},
|
|
||||||
"解析整个CSharp项目": {
|
|
||||||
"Group": "编程",
|
|
||||||
"Color": "stop",
|
|
||||||
"AsButton": False, # 加入下拉菜单中
|
|
||||||
"Info": "解析一个CSharp项目的所有源文件 | 输入参数为路径",
|
|
||||||
"Function": HotReload(解析一个CSharp项目),
|
|
||||||
},
|
|
||||||
"解析Jupyter Notebook文件": {
|
|
||||||
"Group": "编程",
|
|
||||||
"Color": "stop",
|
|
||||||
"AsButton": False,
|
|
||||||
"Info": "解析Jupyter Notebook文件 | 输入参数为路径",
|
|
||||||
"Function": HotReload(解析ipynb文件),
|
|
||||||
"AdvancedArgs": True, # 调用时，唤起高级参数输入区（默认False）
|
|
||||||
"ArgsReminder": "若输入0，则不解析notebook中的Markdown块", # 高级参数输入区的显示提示
|
|
||||||
},
|
|
||||||
"读Tex论文写摘要": {
|
|
||||||
"Group": "学术",
|
|
||||||
"Color": "stop",
|
|
||||||
"AsButton": False,
|
|
||||||
"Info": "读取Tex论文并写摘要 | 输入参数为路径",
|
|
||||||
"Function": HotReload(读文章写摘要),
|
|
||||||
},
|
|
||||||
"翻译README或MD": {
|
|
||||||
"Group": "编程",
|
|
||||||
"Color": "stop",
|
|
||||||
"AsButton": True,
|
|
||||||
"Info": "将Markdown翻译为中文 | 输入参数为路径或URL",
|
|
||||||
"Function": HotReload(Markdown英译中),
|
|
||||||
},
|
|
||||||
"翻译Markdown或README（支持Github链接）": {
|
|
||||||
"Group": "编程",
|
|
||||||
"Color": "stop",
|
|
||||||
"AsButton": False,
|
|
||||||
"Info": "将Markdown或README翻译为中文 | 输入参数为路径或URL",
|
|
||||||
"Function": HotReload(Markdown英译中),
|
|
||||||
},
|
|
||||||
"批量生成函数注释": {
|
|
||||||
"Group": "编程",
|
|
||||||
"Color": "stop",
|
|
||||||
"AsButton": False, # 加入下拉菜单中
|
|
||||||
"Info": "批量生成函数的注释 | 输入参数为路径",
|
|
||||||
"Function": HotReload(批量生成函数注释),
|
|
||||||
},
|
|
||||||
"保存当前的对话": {
|
|
||||||
"Group": "对话",
|
|
||||||
"Color": "stop",
|
|
||||||
"AsButton": True,
|
|
||||||
"Info": "保存当前的对话 | 不需要输入参数",
|
|
||||||
"Function": HotReload(对话历史存档), # 当注册Class后，Function旧接口仅会在“虚空终端”中起作用
|
|
||||||
"Class": Conversation_To_File_Wrap # 新一代插件需要注册Class
|
|
||||||
},
|
|
||||||
"[多线程Demo]解析此项目本身（源码自译解）": {
|
|
||||||
"Group": "对话|编程",
|
|
||||||
"Color": "stop",
|
|
||||||
"AsButton": False, # 加入下拉菜单中
|
|
||||||
"Info": "多线程解析并翻译此项目的源码 | 不需要输入参数",
|
|
||||||
"Function": HotReload(解析项目本身),
|
|
||||||
},
|
|
||||||
"查互联网后回答": {
|
|
||||||
"Group": "对话",
|
|
||||||
"Color": "stop",
|
|
||||||
"AsButton": True, # 加入下拉菜单中
|
|
||||||
# "Info": "连接网络回答问题（需要访问谷歌）| 输入参数是一个问题",
|
|
||||||
"Function": HotReload(连接网络回答问题),
|
|
||||||
"Class": NetworkGPT_Wrap # 新一代插件需要注册Class
|
|
||||||
},
|
|
||||||
"历史上的今天": {
|
|
||||||
"Group": "对话",
|
|
||||||
"Color": "stop",
|
|
||||||
"AsButton": False,
|
|
||||||
"Info": "查看历史上的今天事件 (这是一个面向开发者的插件Demo) | 不需要输入参数",
|
|
||||||
"Function": None,
|
|
||||||
"Class": Demo_Wrap, # 新一代插件需要注册Class
|
|
||||||
},
|
|
||||||
"精准翻译PDF论文": {
|
|
||||||
"Group": "学术",
|
|
||||||
"Color": "stop",
|
|
||||||
"AsButton": True,
|
|
||||||
"Info": "精准翻译PDF论文为中文 | 输入参数为路径",
|
|
||||||
"Function": HotReload(批量翻译PDF文档), # 当注册Class后，Function旧接口仅会在“虚空终端”中起作用
|
|
||||||
"Class": PDF_Tran, # 新一代插件需要注册Class
|
|
||||||
},
|
|
||||||
"询问多个GPT模型": {
|
|
||||||
"Group": "对话",
|
|
||||||
"Color": "stop",
|
|
||||||
"AsButton": True,
|
|
||||||
"Function": HotReload(同时问询),
|
|
||||||
},
|
|
||||||
"批量总结PDF文档": {
|
|
||||||
"Group": "学术",
|
|
||||||
"Color": "stop",
|
|
||||||
"AsButton": False, # 加入下拉菜单中
|
|
||||||
"Info": "批量总结PDF文档的内容 | 输入参数为路径",
|
|
||||||
"Function": HotReload(批量总结PDF文档),
|
|
||||||
},
|
|
||||||
"谷歌学术检索助手（输入谷歌学术搜索页url）": {
|
|
||||||
"Group": "学术",
|
|
||||||
"Color": "stop",
|
|
||||||
"AsButton": False, # 加入下拉菜单中
|
|
||||||
"Info": "使用谷歌学术检索助手搜索指定URL的结果 | 输入参数为谷歌学术搜索页的URL",
|
|
||||||
"Function": HotReload(谷歌检索小助手),
|
|
||||||
},
|
|
||||||
"理解PDF文档内容 （模仿ChatPDF）": {
|
|
||||||
"Group": "学术",
|
|
||||||
"Color": "stop",
|
|
||||||
"AsButton": False, # 加入下拉菜单中
|
|
||||||
"Info": "理解PDF文档的内容并进行回答 | 输入参数为路径",
|
|
||||||
"Function": HotReload(理解PDF文档内容标准文件输入),
|
|
||||||
},
|
|
||||||
"英文Latex项目全文润色（输入路径或上传压缩包）": {
|
|
||||||
"Group": "学术",
|
|
||||||
"Color": "stop",
|
|
||||||
"AsButton": False, # 加入下拉菜单中
|
|
||||||
"Info": "对英文Latex项目全文进行润色处理 | 输入参数为路径或上传压缩包",
|
|
||||||
"Function": HotReload(Latex英文润色),
|
|
||||||
},
|
|
||||||
|
|
||||||
"中文Latex项目全文润色（输入路径或上传压缩包）": {
|
|
||||||
"Group": "学术",
|
|
||||||
"Color": "stop",
|
|
||||||
"AsButton": False, # 加入下拉菜单中
|
|
||||||
"Info": "对中文Latex项目全文进行润色处理 | 输入参数为路径或上传压缩包",
|
|
||||||
"Function": HotReload(Latex中文润色),
|
|
||||||
},
|
|
||||||
# 已经被新插件取代
|
|
||||||
# "英文Latex项目全文纠错（输入路径或上传压缩包）": {
|
|
||||||
# "Group": "学术",
|
|
||||||
# "Color": "stop",
|
|
||||||
# "AsButton": False, # 加入下拉菜单中
|
|
||||||
# "Info": "对英文Latex项目全文进行纠错处理 | 输入参数为路径或上传压缩包",
|
|
||||||
# "Function": HotReload(Latex英文纠错),
|
|
||||||
# },
|
|
||||||
# 已经被新插件取代
|
|
||||||
# "Latex项目全文中译英（输入路径或上传压缩包）": {
|
|
||||||
# "Group": "学术",
|
|
||||||
# "Color": "stop",
|
|
||||||
# "AsButton": False, # 加入下拉菜单中
|
|
||||||
# "Info": "对Latex项目全文进行中译英处理 | 输入参数为路径或上传压缩包",
|
|
||||||
# "Function": HotReload(Latex中译英)
|
|
||||||
# },
|
|
||||||
# 已经被新插件取代
|
|
||||||
# "Latex项目全文英译中（输入路径或上传压缩包）": {
|
|
||||||
# "Group": "学术",
|
|
||||||
# "Color": "stop",
|
|
||||||
# "AsButton": False, # 加入下拉菜单中
|
|
||||||
# "Info": "对Latex项目全文进行英译中处理 | 输入参数为路径或上传压缩包",
|
|
||||||
# "Function": HotReload(Latex英译中)
|
|
||||||
# },
|
|
||||||
"批量Markdown中译英（输入路径或上传压缩包）": {
|
|
||||||
"Group": "编程",
|
|
||||||
"Color": "stop",
|
|
||||||
"AsButton": False, # 加入下拉菜单中
|
|
||||||
"Info": "批量将Markdown文件中文翻译为英文 | 输入参数为路径或上传压缩包",
|
|
||||||
"Function": HotReload(Markdown中译英),
|
|
||||||
},
|
|
||||||
"Latex英文纠错+高亮修正位置 [需Latex]": {
|
|
||||||
"Group": "学术",
|
|
||||||
"Color": "stop",
|
|
||||||
"AsButton": False,
|
|
||||||
"AdvancedArgs": True,
|
|
||||||
"ArgsReminder": "如果有必要, 请在此处追加更细致的矫错指令（使用英文）。",
|
|
||||||
"Function": HotReload(Latex英文纠错加PDF对比),
|
|
||||||
},
|
|
||||||
"📚Arxiv论文精细翻译（输入arxivID）[需Latex]": {
|
|
||||||
"Group": "学术",
|
|
||||||
"Color": "stop",
|
|
||||||
"AsButton": False,
|
|
||||||
"AdvancedArgs": True,
|
|
||||||
"ArgsReminder": r"如果有必要, 请在此处给出自定义翻译命令, 解决部分词汇翻译不准确的问题。 "
|
|
||||||
r"例如当单词'agent'翻译不准确时, 请尝试把以下指令复制到高级参数区: "
|
|
||||||
r'If the term "agent" is used in this section, it should be translated to "智能体". ',
|
|
||||||
"Info": "Arixv论文精细翻译 | 输入参数arxiv论文的ID，比如1812.10695",
|
|
||||||
"Function": HotReload(Latex翻译中文并重新编译PDF), # 当注册Class后，Function旧接口仅会在“虚空终端”中起作用
|
|
||||||
"Class": Arxiv_Localize, # 新一代插件需要注册Class
|
|
||||||
},
|
|
||||||
"📚本地Latex论文精细翻译（上传Latex项目）[需Latex]": {
|
|
||||||
"Group": "学术",
|
|
||||||
"Color": "stop",
|
|
||||||
"AsButton": False,
|
|
||||||
"AdvancedArgs": True,
|
|
||||||
"ArgsReminder": r"如果有必要, 请在此处给出自定义翻译命令, 解决部分词汇翻译不准确的问题。 "
|
|
||||||
r"例如当单词'agent'翻译不准确时, 请尝试把以下指令复制到高级参数区: "
|
|
||||||
r'If the term "agent" is used in this section, it should be translated to "智能体". ',
|
|
||||||
"Info": "本地Latex论文精细翻译 | 输入参数是路径",
|
|
||||||
"Function": HotReload(Latex翻译中文并重新编译PDF),
|
|
||||||
},
|
|
||||||
"PDF翻译中文并重新编译PDF（上传PDF）[需Latex]": {
|
|
||||||
"Group": "学术",
|
|
||||||
"Color": "stop",
|
|
||||||
"AsButton": False,
|
|
||||||
"AdvancedArgs": True,
|
|
||||||
"ArgsReminder": r"如果有必要, 请在此处给出自定义翻译命令, 解决部分词汇翻译不准确的问题。 "
|
|
||||||
r"例如当单词'agent'翻译不准确时, 请尝试把以下指令复制到高级参数区: "
|
|
||||||
r'If the term "agent" is used in this section, it should be translated to "智能体". ',
|
|
||||||
"Info": "PDF翻译中文，并重新编译PDF | 输入参数为路径",
|
|
||||||
"Function": HotReload(PDF翻译中文并重新编译PDF), # 当注册Class后，Function旧接口仅会在“虚空终端”中起作用
|
|
||||||
"Class": PDF_Localize # 新一代插件需要注册Class
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
function_plugins.update(
|
|
||||||
{
|
|
||||||
"🎨图片生成（DALLE2/DALLE3, 使用前切换到GPT系列模型）": {
|
|
||||||
"Group": "对话",
|
|
||||||
"Color": "stop",
|
|
||||||
"AsButton": False,
|
|
||||||
"Info": "使用 DALLE2/DALLE3 生成图片 | 输入参数字符串，提供图像的内容",
|
|
||||||
"Function": HotReload(图片生成_DALLE2), # 当注册Class后，Function旧接口仅会在“虚空终端”中起作用
|
|
||||||
"Class": ImageGen_Wrap # 新一代插件需要注册Class
|
|
||||||
},
|
|
||||||
}
|
|
||||||
)
|
|
||||||
|
|
||||||
function_plugins.update(
|
|
||||||
{
|
|
||||||
"🎨图片修改_DALLE2 （使用前请切换模型到GPT系列）": {
|
|
||||||
"Group": "对话",
|
|
||||||
"Color": "stop",
|
|
||||||
"AsButton": False,
|
|
||||||
"AdvancedArgs": False, # 调用时，唤起高级参数输入区（默认False）
|
|
||||||
# "Info": "使用DALLE2修改图片 | 输入参数字符串，提供图像的内容",
|
|
||||||
"Function": HotReload(图片修改_DALLE2),
|
|
||||||
},
|
|
||||||
}
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
# -=--=- 尚未充分测试的实验性插件 & 需要额外依赖的插件 -=--=-
|
|
||||||
try:
|
|
||||||
from crazy_functions.下载arxiv论文翻译摘要 import 下载arxiv论文并翻译摘要
|
|
||||||
|
|
||||||
function_plugins.update(
|
|
||||||
{
|
|
||||||
"一键下载arxiv论文并翻译摘要（先在input输入编号，如1812.10695）": {
|
|
||||||
"Group": "学术",
|
|
||||||
"Color": "stop",
|
|
||||||
"AsButton": False, # 加入下拉菜单中
|
|
||||||
# "Info": "下载arxiv论文并翻译摘要 | 输入参数为arxiv编号如1812.10695",
|
|
||||||
"Function": HotReload(下载arxiv论文并翻译摘要),
|
|
||||||
}
|
|
||||||
}
|
|
||||||
)
|
|
||||||
except:
|
|
||||||
logger.error(trimmed_format_exc())
|
|
||||||
logger.error("Load function plugin failed")
|
|
||||||
|
|
||||||
# try:
|
|
||||||
# from crazy_functions.联网的ChatGPT import 连接网络回答问题
|
|
||||||
|
|
||||||
# function_plugins.update(
|
|
||||||
# {
|
|
||||||
# "连接网络回答问题（输入问题后点击该插件，需要访问谷歌）": {
|
|
||||||
# "Group": "对话",
|
|
||||||
# "Color": "stop",
|
|
||||||
# "AsButton": False, # 加入下拉菜单中
|
|
||||||
# # "Info": "连接网络回答问题（需要访问谷歌）| 输入参数是一个问题",
|
|
||||||
# "Function": HotReload(连接网络回答问题),
|
|
||||||
# }
|
|
||||||
# }
|
|
||||||
# )
|
|
||||||
# from crazy_functions.联网的ChatGPT_bing版 import 连接bing搜索回答问题
|
|
||||||
|
|
||||||
# function_plugins.update(
|
|
||||||
# {
|
|
||||||
# "连接网络回答问题（中文Bing版，输入问题后点击该插件）": {
|
|
||||||
# "Group": "对话",
|
|
||||||
# "Color": "stop",
|
|
||||||
# "AsButton": False, # 加入下拉菜单中
|
|
||||||
# "Info": "连接网络回答问题（需要访问中文Bing）| 输入参数是一个问题",
|
|
||||||
# "Function": HotReload(连接bing搜索回答问题),
|
|
||||||
# }
|
|
||||||
# }
|
|
||||||
# )
|
|
||||||
# except:
|
|
||||||
# logger.error(trimmed_format_exc())
|
|
||||||
# logger.error("Load function plugin failed")
|
|
||||||
|
|
||||||
try:
|
|
||||||
from crazy_functions.SourceCode_Analyse import 解析任意code项目
|
|
||||||
|
|
||||||
function_plugins.update(
|
|
||||||
{
|
|
||||||
"解析项目源代码（手动指定和筛选源代码文件类型）": {
|
|
||||||
"Group": "编程",
|
|
||||||
"Color": "stop",
|
|
||||||
"AsButton": False,
|
|
||||||
"AdvancedArgs": True, # 调用时，唤起高级参数输入区（默认False）
|
|
||||||
"ArgsReminder": '输入时用逗号隔开, *代表通配符, 加了^代表不匹配; 不输入代表全部匹配。例如: "*.c, ^*.cpp, config.toml, ^*.toml"', # 高级参数输入区的显示提示
|
|
||||||
"Function": HotReload(解析任意code项目),
|
|
||||||
},
|
|
||||||
}
|
|
||||||
)
|
|
||||||
except:
|
|
||||||
logger.error(trimmed_format_exc())
|
|
||||||
logger.error("Load function plugin failed")
|
|
||||||
|
|
||||||
try:
|
|
||||||
from crazy_functions.询问多个大语言模型 import 同时问询_指定模型
|
|
||||||
|
|
||||||
function_plugins.update(
|
|
||||||
{
|
|
||||||
"询问多个GPT模型（手动指定询问哪些模型）": {
|
|
||||||
"Group": "对话",
|
|
||||||
"Color": "stop",
|
|
||||||
"AsButton": False,
|
|
||||||
"AdvancedArgs": True, # 调用时，唤起高级参数输入区（默认False）
|
|
||||||
"ArgsReminder": "支持任意数量的llm接口，用&符号分隔。例如chatglm&gpt-3.5-turbo&gpt-4", # 高级参数输入区的显示提示
|
|
||||||
"Function": HotReload(同时问询_指定模型),
|
|
||||||
},
|
|
||||||
}
|
|
||||||
)
|
|
||||||
except:
|
|
||||||
logger.error(trimmed_format_exc())
|
|
||||||
logger.error("Load function plugin failed")
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
try:
|
|
||||||
from crazy_functions.总结音视频 import 总结音视频
|
|
||||||
|
|
||||||
function_plugins.update(
|
|
||||||
{
|
|
||||||
"批量总结音视频（输入路径或上传压缩包）": {
|
|
||||||
"Group": "对话",
|
|
||||||
"Color": "stop",
|
|
||||||
"AsButton": False,
|
|
||||||
"AdvancedArgs": True,
|
|
||||||
"ArgsReminder": "调用openai api 使用whisper-1模型, 目前支持的格式:mp4, m4a, wav, mpga, mpeg, mp3。此处可以输入解析提示，例如：解析为简体中文（默认）。",
|
|
||||||
"Info": "批量总结音频或视频 | 输入参数为路径",
|
|
||||||
"Function": HotReload(总结音视频),
|
|
||||||
}
|
|
||||||
}
|
|
||||||
)
|
|
||||||
except:
|
|
||||||
logger.error(trimmed_format_exc())
|
|
||||||
logger.error("Load function plugin failed")
|
|
||||||
|
|
||||||
try:
|
|
||||||
from crazy_functions.数学动画生成manim import 动画生成
|
|
||||||
|
|
||||||
function_plugins.update(
|
|
||||||
{
|
|
||||||
"数学动画生成（Manim）": {
|
|
||||||
"Group": "对话",
|
|
||||||
"Color": "stop",
|
|
||||||
"AsButton": False,
|
|
||||||
"Info": "按照自然语言描述生成一个动画 | 输入参数是一段话",
|
|
||||||
"Function": HotReload(动画生成),
|
|
||||||
}
|
|
||||||
}
|
|
||||||
)
|
|
||||||
except:
|
|
||||||
logger.error(trimmed_format_exc())
|
|
||||||
logger.error("Load function plugin failed")
|
|
||||||
|
|
||||||
try:
|
|
||||||
from crazy_functions.Markdown_Translate import Markdown翻译指定语言
|
|
||||||
|
|
||||||
function_plugins.update(
|
|
||||||
{
|
|
||||||
"Markdown翻译（指定翻译成何种语言）": {
|
|
||||||
"Group": "编程",
|
|
||||||
"Color": "stop",
|
|
||||||
"AsButton": False,
|
|
||||||
"AdvancedArgs": True,
|
|
||||||
"ArgsReminder": "请输入要翻译成哪种语言，默认为Chinese。",
|
|
||||||
"Function": HotReload(Markdown翻译指定语言),
|
|
||||||
}
|
|
||||||
}
|
|
||||||
)
|
|
||||||
except:
|
|
||||||
logger.error(trimmed_format_exc())
|
|
||||||
logger.error("Load function plugin failed")
|
|
||||||
|
|
||||||
try:
|
|
||||||
from crazy_functions.知识库问答 import 知识库文件注入
|
|
||||||
|
|
||||||
function_plugins.update(
|
|
||||||
{
|
|
||||||
"构建知识库（先上传文件素材,再运行此插件）": {
|
|
||||||
"Group": "对话",
|
|
||||||
"Color": "stop",
|
|
||||||
"AsButton": False,
|
|
||||||
"AdvancedArgs": True,
|
|
||||||
"ArgsReminder": "此处待注入的知识库名称id, 默认为default。文件进入知识库后可长期保存。可以通过再次调用本插件的方式，向知识库追加更多文档。",
|
|
||||||
"Function": HotReload(知识库文件注入),
|
|
||||||
}
|
|
||||||
}
|
|
||||||
)
|
|
||||||
except:
|
|
||||||
logger.error(trimmed_format_exc())
|
|
||||||
logger.error("Load function plugin failed")
|
|
||||||
|
|
||||||
try:
|
|
||||||
from crazy_functions.知识库问答 import 读取知识库作答
|
|
||||||
|
|
||||||
function_plugins.update(
|
|
||||||
{
|
|
||||||
"知识库文件注入（构建知识库后,再运行此插件）": {
|
|
||||||
"Group": "对话",
|
|
||||||
"Color": "stop",
|
|
||||||
"AsButton": False,
|
|
||||||
"AdvancedArgs": True,
|
|
||||||
"ArgsReminder": "待提取的知识库名称id, 默认为default, 您需要构建知识库后再运行此插件。",
|
|
||||||
"Function": HotReload(读取知识库作答),
|
|
||||||
}
|
|
||||||
}
|
|
||||||
)
|
|
||||||
except:
|
|
||||||
logger.error(trimmed_format_exc())
|
|
||||||
logger.error("Load function plugin failed")
|
|
||||||
|
|
||||||
try:
|
|
||||||
from crazy_functions.交互功能函数模板 import 交互功能模板函数
|
|
||||||
|
|
||||||
function_plugins.update(
|
|
||||||
{
|
|
||||||
"交互功能模板Demo函数（查找wallhaven.cc的壁纸）": {
|
|
||||||
"Group": "对话",
|
|
||||||
"Color": "stop",
|
|
||||||
"AsButton": False,
|
|
||||||
"Function": HotReload(交互功能模板函数),
|
|
||||||
}
|
|
||||||
}
|
|
||||||
)
|
|
||||||
except:
|
|
||||||
logger.error(trimmed_format_exc())
|
|
||||||
logger.error("Load function plugin failed")
|
|
||||||
|
|
||||||
|
|
||||||
try:
|
|
||||||
from toolbox import get_conf
|
|
||||||
|
|
||||||
ENABLE_AUDIO = get_conf("ENABLE_AUDIO")
|
|
||||||
if ENABLE_AUDIO:
|
|
||||||
from crazy_functions.语音助手 import 语音助手
|
|
||||||
|
|
||||||
function_plugins.update(
|
|
||||||
{
|
|
||||||
"实时语音对话": {
|
|
||||||
"Group": "对话",
|
|
||||||
"Color": "stop",
|
|
||||||
"AsButton": True,
|
|
||||||
"Info": "这是一个时刻聆听着的语音对话助手 | 没有输入参数",
|
|
||||||
"Function": HotReload(语音助手),
|
|
||||||
}
|
|
||||||
}
|
|
||||||
)
|
|
||||||
except:
|
|
||||||
logger.error(trimmed_format_exc())
|
|
||||||
logger.error("Load function plugin failed")
|
|
||||||
|
|
||||||
try:
|
|
||||||
from crazy_functions.批量翻译PDF文档_NOUGAT import 批量翻译PDF文档
|
|
||||||
|
|
||||||
function_plugins.update(
|
|
||||||
{
|
|
||||||
"精准翻译PDF文档（NOUGAT）": {
|
|
||||||
"Group": "学术",
|
|
||||||
"Color": "stop",
|
|
||||||
"AsButton": False,
|
|
||||||
"Function": HotReload(批量翻译PDF文档),
|
|
||||||
}
|
|
||||||
}
|
|
||||||
)
|
|
||||||
except:
|
|
||||||
logger.error(trimmed_format_exc())
|
|
||||||
logger.error("Load function plugin failed")
|
|
||||||
|
|
||||||
try:
|
|
||||||
from crazy_functions.函数动态生成 import 函数动态生成
|
|
||||||
|
|
||||||
function_plugins.update(
|
|
||||||
{
|
|
||||||
"动态代码解释器（CodeInterpreter）": {
|
|
||||||
"Group": "智能体",
|
|
||||||
"Color": "stop",
|
|
||||||
"AsButton": False,
|
|
||||||
"Function": HotReload(函数动态生成),
|
|
||||||
}
|
|
||||||
}
|
|
||||||
)
|
|
||||||
except:
|
|
||||||
logger.error(trimmed_format_exc())
|
|
||||||
logger.error("Load function plugin failed")
|
|
||||||
|
|
||||||
try:
|
|
||||||
from crazy_functions.多智能体 import 多智能体终端
|
|
||||||
|
|
||||||
function_plugins.update(
|
|
||||||
{
|
|
||||||
"AutoGen多智能体终端（仅供测试）": {
|
|
||||||
"Group": "智能体",
|
|
||||||
"Color": "stop",
|
|
||||||
"AsButton": False,
|
|
||||||
"Function": HotReload(多智能体终端),
|
|
||||||
}
|
|
||||||
}
|
|
||||||
)
|
|
||||||
except:
|
|
||||||
logger.error(trimmed_format_exc())
|
|
||||||
logger.error("Load function plugin failed")
|
|
||||||
|
|
||||||
try:
|
|
||||||
from crazy_functions.互动小游戏 import 随机小游戏
|
|
||||||
|
|
||||||
function_plugins.update(
|
|
||||||
{
|
|
||||||
"随机互动小游戏（仅供测试）": {
|
|
||||||
"Group": "智能体",
|
|
||||||
"Color": "stop",
|
|
||||||
"AsButton": False,
|
|
||||||
"Function": HotReload(随机小游戏),
|
|
||||||
}
|
|
||||||
}
|
|
||||||
)
|
|
||||||
except:
|
|
||||||
logger.error(trimmed_format_exc())
|
|
||||||
logger.error("Load function plugin failed")
|
|
||||||
|
|
||||||
try:
|
|
||||||
from crazy_functions.Rag_Interface import Rag问答
|
|
||||||
|
|
||||||
function_plugins.update(
|
|
||||||
{
|
|
||||||
"Rag智能召回": {
|
|
||||||
"Group": "对话",
|
|
||||||
"Color": "stop",
|
|
||||||
"AsButton": False,
|
|
||||||
"Info": "将问答数据记录到向量库中，作为长期参考。",
|
|
||||||
"Function": HotReload(Rag问答),
|
|
||||||
},
|
|
||||||
}
|
|
||||||
)
|
|
||||||
except:
|
|
||||||
logger.error(trimmed_format_exc())
|
|
||||||
logger.error("Load function plugin failed")
|
|
||||||
|
|
||||||
|
|
||||||
# try:
|
|
||||||
# from crazy_functions.高级功能函数模板 import 测试图表渲染
|
|
||||||
# function_plugins.update({
|
|
||||||
# "绘制逻辑关系（测试图表渲染）": {
|
|
||||||
# "Group": "智能体",
|
|
||||||
# "Color": "stop",
|
|
||||||
# "AsButton": True,
|
|
||||||
# "Function": HotReload(测试图表渲染)
|
|
||||||
# }
|
|
||||||
# })
|
|
||||||
# except:
|
|
||||||
# logger.error(trimmed_format_exc())
|
|
||||||
# print('Load function plugin failed')
|
|
||||||
|
|
||||||
|
|
||||||
"""
|
"""
|
||||||
设置默认值:
|
设置默认值:
|
||||||
@@ -762,8 +36,6 @@ def get_crazy_functions():
|
|||||||
return function_plugins
|
return function_plugins
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
def get_multiplex_button_functions():
|
def get_multiplex_button_functions():
|
||||||
"""多路复用主提交按钮的功能映射
|
"""多路复用主提交按钮的功能映射
|
||||||
"""
|
"""
|
||||||
@@ -771,12 +43,6 @@ def get_multiplex_button_functions():
|
|||||||
"常规对话":
|
"常规对话":
|
||||||
"",
|
"",
|
||||||
|
|
||||||
"多模型对话":
|
"蚂小财测试":
|
||||||
"询问多个GPT模型", # 映射到上面的 `询问多个GPT模型` 插件
|
"蚂小财测试", # 映射到上面的 `询问多个GPT模型` 插件
|
||||||
|
|
||||||
"智能召回 RAG":
|
|
||||||
"Rag智能召回", # 映射到上面的 `Rag智能召回` 插件
|
|
||||||
|
|
||||||
"多媒体查询":
|
|
||||||
"多媒体智能体", # 映射到上面的 `多媒体智能体` 插件
|
|
||||||
}
|
}
|
||||||
|
|||||||
9
crazy_functions/AntFin.py
Normal file
9
crazy_functions/AntFin.py
Normal file
@@ -0,0 +1,9 @@
|
|||||||
|
from toolbox import CatchException, update_ui, get_conf, select_api_key, get_log_folder
|
||||||
|
from crazy_functions.multi_stage.multi_stage_utils import GptAcademicState
|
||||||
|
|
||||||
|
|
||||||
|
@CatchException
|
||||||
|
def AntFinTest(prompt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_request):
|
||||||
|
chatbot.append(("AntFin Test", "AntFin Test"))
|
||||||
|
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面 界面更新
|
||||||
|
|
||||||
@@ -1,7 +1,7 @@
|
|||||||
|
|
||||||
from toolbox import get_conf, update_ui
|
from toolbox import get_conf, update_ui
|
||||||
from crazy_functions.Image_Generate import 图片生成_DALLE2, 图片生成_DALLE3, 图片修改_DALLE2
|
|
||||||
from crazy_functions.plugin_template.plugin_class_template import GptAcademicPluginTemplate, ArgProperty
|
from crazy_functions.plugin_template.plugin_class_template import GptAcademicPluginTemplate, ArgProperty
|
||||||
|
from crazy_functions.AntFin import AntFinTest
|
||||||
|
|
||||||
|
|
||||||
class ImageGen_Wrap(GptAcademicPluginTemplate):
|
class ImageGen_Wrap(GptAcademicPluginTemplate):
|
||||||
@@ -38,19 +38,6 @@ class ImageGen_Wrap(GptAcademicPluginTemplate):
|
|||||||
"""
|
"""
|
||||||
执行插件
|
执行插件
|
||||||
"""
|
"""
|
||||||
# 分辨率
|
yield from AntFinTest(txt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_request)
|
||||||
resolution = plugin_kwargs["resolution"].replace("(限DALLE2)", "").replace("(限DALLE3)", "")
|
|
||||||
|
|
||||||
if plugin_kwargs["model_name"] == "DALLE2":
|
|
||||||
plugin_kwargs["advanced_arg"] = resolution
|
|
||||||
yield from 图片生成_DALLE2(txt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_request)
|
|
||||||
|
|
||||||
elif plugin_kwargs["model_name"] == "DALLE3":
|
|
||||||
quality = plugin_kwargs["quality (仅DALLE3生效)"]
|
|
||||||
style = plugin_kwargs["style (仅DALLE3生效)"]
|
|
||||||
plugin_kwargs["advanced_arg"] = f"{resolution}-{quality}-{style}"
|
|
||||||
yield from 图片生成_DALLE3(txt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_request)
|
|
||||||
|
|
||||||
else:
|
|
||||||
chatbot.append([None, "抱歉，找不到该模型"])
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
@@ -1,220 +0,0 @@
|
|||||||
from toolbox import CatchException, update_ui, promote_file_to_downloadzone, get_log_folder, get_user
|
|
||||||
from crazy_functions.plugin_template.plugin_class_template import GptAcademicPluginTemplate, ArgProperty
|
|
||||||
import re
|
|
||||||
|
|
||||||
f_prefix = 'GPT-Academic对话存档'
|
|
||||||
|
|
||||||
def write_chat_to_file(chatbot, history=None, file_name=None):
|
|
||||||
"""
|
|
||||||
将对话记录history以Markdown格式写入文件中。如果没有指定文件名，则使用当前时间生成文件名。
|
|
||||||
"""
|
|
||||||
import os
|
|
||||||
import time
|
|
||||||
from themes.theme import advanced_css
|
|
||||||
|
|
||||||
if file_name is None:
|
|
||||||
file_name = f_prefix + time.strftime("%Y-%m-%d-%H-%M-%S", time.localtime()) + '.html'
|
|
||||||
fp = os.path.join(get_log_folder(get_user(chatbot), plugin_name='chat_history'), file_name)
|
|
||||||
|
|
||||||
with open(fp, 'w', encoding='utf8') as f:
|
|
||||||
from textwrap import dedent
|
|
||||||
form = dedent("""
|
|
||||||
<!DOCTYPE html><head><meta charset="utf-8"><title>对话存档</title><style>{CSS}</style></head>
|
|
||||||
<body>
|
|
||||||
<div class="test_temp1" style="width:10%; height: 500px; float:left;"></div>
|
|
||||||
<div class="test_temp2" style="width:80%;padding: 40px;float:left;padding-left: 20px;padding-right: 20px;box-shadow: rgba(0, 0, 0, 0.2) 0px 0px 8px 8px;border-radius: 10px;">
|
|
||||||
<div class="chat-body" style="display: flex;justify-content: center;flex-direction: column;align-items: center;flex-wrap: nowrap;">
|
|
||||||
{CHAT_PREVIEW}
|
|
||||||
<div></div>
|
|
||||||
<div></div>
|
|
||||||
<div style="text-align: center;width:80%;padding: 0px;float:left;padding-left:20px;padding-right:20px;box-shadow: rgba(0, 0, 0, 0.05) 0px 0px 1px 2px;border-radius: 1px;">对话（原始数据）</div>
|
|
||||||
{HISTORY_PREVIEW}
|
|
||||||
</div>
|
|
||||||
</div>
|
|
||||||
<div class="test_temp3" style="width:10%; height: 500px; float:left;"></div>
|
|
||||||
</body>
|
|
||||||
""")
|
|
||||||
|
|
||||||
qa_from = dedent("""
|
|
||||||
<div class="QaBox" style="width:80%;padding: 20px;margin-bottom: 20px;box-shadow: rgb(0 255 159 / 50%) 0px 0px 1px 2px;border-radius: 4px;">
|
|
||||||
<div class="Question" style="border-radius: 2px;">{QUESTION}</div>
|
|
||||||
<hr color="blue" style="border-top: dotted 2px #ccc;">
|
|
||||||
<div class="Answer" style="border-radius: 2px;">{ANSWER}</div>
|
|
||||||
</div>
|
|
||||||
""")
|
|
||||||
|
|
||||||
history_from = dedent("""
|
|
||||||
<div class="historyBox" style="width:80%;padding: 0px;float:left;padding-left:20px;padding-right:20px;box-shadow: rgba(0, 0, 0, 0.05) 0px 0px 1px 2px;border-radius: 1px;">
|
|
||||||
<div class="entry" style="border-radius: 2px;">{ENTRY}</div>
|
|
||||||
</div>
|
|
||||||
""")
|
|
||||||
CHAT_PREVIEW_BUF = ""
|
|
||||||
for i, contents in enumerate(chatbot):
|
|
||||||
question, answer = contents[0], contents[1]
|
|
||||||
if question is None: question = ""
|
|
||||||
try: question = str(question)
|
|
||||||
except: question = ""
|
|
||||||
if answer is None: answer = ""
|
|
||||||
try: answer = str(answer)
|
|
||||||
except: answer = ""
|
|
||||||
CHAT_PREVIEW_BUF += qa_from.format(QUESTION=question, ANSWER=answer)
|
|
||||||
|
|
||||||
HISTORY_PREVIEW_BUF = ""
|
|
||||||
for h in history:
|
|
||||||
HISTORY_PREVIEW_BUF += history_from.format(ENTRY=h)
|
|
||||||
html_content = form.format(CHAT_PREVIEW=CHAT_PREVIEW_BUF, HISTORY_PREVIEW=HISTORY_PREVIEW_BUF, CSS=advanced_css)
|
|
||||||
f.write(html_content)
|
|
||||||
|
|
||||||
promote_file_to_downloadzone(fp, rename_file=file_name, chatbot=chatbot)
|
|
||||||
return '对话历史写入：' + fp
|
|
||||||
|
|
||||||
def gen_file_preview(file_name):
|
|
||||||
try:
|
|
||||||
with open(file_name, 'r', encoding='utf8') as f:
|
|
||||||
file_content = f.read()
|
|
||||||
# pattern to match the text between <head> and </head>
|
|
||||||
pattern = re.compile(r'<head>.*?</head>', flags=re.DOTALL)
|
|
||||||
file_content = re.sub(pattern, '', file_content)
|
|
||||||
html, history = file_content.split('<hr color="blue"> \n\n 对话数据 (无渲染):\n')
|
|
||||||
history = history.strip('<code>')
|
|
||||||
history = history.strip('</code>')
|
|
||||||
history = history.split("\n>>>")
|
|
||||||
return list(filter(lambda x:x!="", history))[0][:100]
|
|
||||||
except:
|
|
||||||
return ""
|
|
||||||
|
|
||||||
def read_file_to_chat(chatbot, history, file_name):
|
|
||||||
with open(file_name, 'r', encoding='utf8') as f:
|
|
||||||
file_content = f.read()
|
|
||||||
from bs4 import BeautifulSoup
|
|
||||||
soup = BeautifulSoup(file_content, 'lxml')
|
|
||||||
# 提取QaBox信息
|
|
||||||
chatbot.clear()
|
|
||||||
qa_box_list = []
|
|
||||||
qa_boxes = soup.find_all("div", class_="QaBox")
|
|
||||||
for box in qa_boxes:
|
|
||||||
question = box.find("div", class_="Question").get_text(strip=False)
|
|
||||||
answer = box.find("div", class_="Answer").get_text(strip=False)
|
|
||||||
qa_box_list.append({"Question": question, "Answer": answer})
|
|
||||||
chatbot.append([question, answer])
|
|
||||||
# 提取historyBox信息
|
|
||||||
history_box_list = []
|
|
||||||
history_boxes = soup.find_all("div", class_="historyBox")
|
|
||||||
for box in history_boxes:
|
|
||||||
entry = box.find("div", class_="entry").get_text(strip=False)
|
|
||||||
history_box_list.append(entry)
|
|
||||||
history = history_box_list
|
|
||||||
chatbot.append([None, f"[Local Message] 载入对话{len(qa_box_list)}条，上下文{len(history)}条。"])
|
|
||||||
return chatbot, history
|
|
||||||
|
|
||||||
@CatchException
|
|
||||||
def 对话历史存档(txt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_request):
|
|
||||||
"""
|
|
||||||
txt 输入栏用户输入的文本，例如需要翻译的一段话，再例如一个包含了待处理文件的路径
|
|
||||||
llm_kwargs gpt模型参数，如温度和top_p等，一般原样传递下去就行
|
|
||||||
plugin_kwargs 插件模型的参数，暂时没有用武之地
|
|
||||||
chatbot 聊天显示框的句柄，用于显示给用户
|
|
||||||
history 聊天历史，前情提要
|
|
||||||
system_prompt 给gpt的静默提醒
|
|
||||||
user_request 当前用户的请求信息（IP地址等）
|
|
||||||
"""
|
|
||||||
file_name = plugin_kwargs.get("file_name", None)
|
|
||||||
if (file_name is not None) and (file_name != "") and (not file_name.endswith('.html')): file_name += '.html'
|
|
||||||
else: file_name = None
|
|
||||||
|
|
||||||
chatbot.append((None, f"[Local Message] {write_chat_to_file(chatbot, history, file_name)}，您可以调用下拉菜单中的“载入对话历史存档”还原当下的对话。"))
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面 # 由于请求gpt需要一段时间，我们先及时地做一次界面更新
|
|
||||||
|
|
||||||
|
|
||||||
class Conversation_To_File_Wrap(GptAcademicPluginTemplate):
|
|
||||||
def __init__(self):
|
|
||||||
"""
|
|
||||||
请注意`execute`会执行在不同的线程中，因此您在定义和使用类变量时，应当慎之又慎！
|
|
||||||
"""
|
|
||||||
pass
|
|
||||||
|
|
||||||
def define_arg_selection_menu(self):
|
|
||||||
"""
|
|
||||||
定义插件的二级选项菜单
|
|
||||||
|
|
||||||
第一个参数，名称`file_name`，参数`type`声明这是一个文本框，文本框上方显示`title`，文本框内部显示`description`，`default_value`为默认值；
|
|
||||||
"""
|
|
||||||
gui_definition = {
|
|
||||||
"file_name": ArgProperty(title="保存文件名", description="输入对话存档文件名，留空则使用时间作为文件名", default_value="", type="string").model_dump_json(), # 主输入，自动从输入框同步
|
|
||||||
}
|
|
||||||
return gui_definition
|
|
||||||
|
|
||||||
def execute(txt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_request):
|
|
||||||
"""
|
|
||||||
执行插件
|
|
||||||
"""
|
|
||||||
yield from 对话历史存档(txt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_request)
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
def hide_cwd(str):
|
|
||||||
import os
|
|
||||||
current_path = os.getcwd()
|
|
||||||
replace_path = "."
|
|
||||||
return str.replace(current_path, replace_path)
|
|
||||||
|
|
||||||
@CatchException
|
|
||||||
def 载入对话历史存档(txt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_request):
|
|
||||||
"""
|
|
||||||
txt 输入栏用户输入的文本，例如需要翻译的一段话，再例如一个包含了待处理文件的路径
|
|
||||||
llm_kwargs gpt模型参数，如温度和top_p等，一般原样传递下去就行
|
|
||||||
plugin_kwargs 插件模型的参数，暂时没有用武之地
|
|
||||||
chatbot 聊天显示框的句柄，用于显示给用户
|
|
||||||
history 聊天历史，前情提要
|
|
||||||
system_prompt 给gpt的静默提醒
|
|
||||||
user_request 当前用户的请求信息（IP地址等）
|
|
||||||
"""
|
|
||||||
from crazy_functions.crazy_utils import get_files_from_everything
|
|
||||||
success, file_manifest, _ = get_files_from_everything(txt, type='.html')
|
|
||||||
|
|
||||||
if not success:
|
|
||||||
if txt == "": txt = '空空如也的输入栏'
|
|
||||||
import glob
|
|
||||||
local_history = "<br/>".join([
|
|
||||||
"`"+hide_cwd(f)+f" ({gen_file_preview(f)})"+"`"
|
|
||||||
for f in glob.glob(
|
|
||||||
f'{get_log_folder(get_user(chatbot), plugin_name="chat_history")}/**/{f_prefix}*.html',
|
|
||||||
recursive=True
|
|
||||||
)])
|
|
||||||
chatbot.append([f"正在查找对话历史文件（html格式）: {txt}", f"找不到任何html文件: {txt}。但本地存储了以下历史文件，您可以将任意一个文件路径粘贴到输入区，然后重试：<br/>{local_history}"])
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
|
|
||||||
try:
|
|
||||||
chatbot, history = read_file_to_chat(chatbot, history, file_manifest[0])
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
except:
|
|
||||||
chatbot.append([f"载入对话历史文件", f"对话历史文件损坏！"])
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
|
|
||||||
@CatchException
|
|
||||||
def 删除所有本地对话历史记录(txt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_request):
|
|
||||||
"""
|
|
||||||
txt 输入栏用户输入的文本，例如需要翻译的一段话，再例如一个包含了待处理文件的路径
|
|
||||||
llm_kwargs gpt模型参数，如温度和top_p等，一般原样传递下去就行
|
|
||||||
plugin_kwargs 插件模型的参数，暂时没有用武之地
|
|
||||||
chatbot 聊天显示框的句柄，用于显示给用户
|
|
||||||
history 聊天历史，前情提要
|
|
||||||
system_prompt 给gpt的静默提醒
|
|
||||||
user_request 当前用户的请求信息（IP地址等）
|
|
||||||
"""
|
|
||||||
|
|
||||||
import glob, os
|
|
||||||
local_history = "<br/>".join([
|
|
||||||
"`"+hide_cwd(f)+"`"
|
|
||||||
for f in glob.glob(
|
|
||||||
f'{get_log_folder(get_user(chatbot), plugin_name="chat_history")}/**/{f_prefix}*.html', recursive=True
|
|
||||||
)])
|
|
||||||
for f in glob.glob(f'{get_log_folder(get_user(chatbot), plugin_name="chat_history")}/**/{f_prefix}*.html', recursive=True):
|
|
||||||
os.remove(f)
|
|
||||||
chatbot.append([f"删除所有历史对话文件", f"已删除<br/>{local_history}"])
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
@@ -1,276 +0,0 @@
|
|||||||
from toolbox import CatchException, update_ui, get_conf, select_api_key, get_log_folder
|
|
||||||
from crazy_functions.multi_stage.multi_stage_utils import GptAcademicState
|
|
||||||
|
|
||||||
|
|
||||||
def gen_image(llm_kwargs, prompt, resolution="1024x1024", model="dall-e-2", quality=None, style=None):
|
|
||||||
import requests, json, time, os
|
|
||||||
from request_llms.bridge_all import model_info
|
|
||||||
|
|
||||||
proxies = get_conf('proxies')
|
|
||||||
# Set up OpenAI API key and model
|
|
||||||
api_key = select_api_key(llm_kwargs['api_key'], llm_kwargs['llm_model'])
|
|
||||||
chat_endpoint = model_info[llm_kwargs['llm_model']]['endpoint']
|
|
||||||
# 'https://api.openai.com/v1/chat/completions'
|
|
||||||
img_endpoint = chat_endpoint.replace('chat/completions','images/generations')
|
|
||||||
# # Generate the image
|
|
||||||
url = img_endpoint
|
|
||||||
headers = {
|
|
||||||
'Authorization': f"Bearer {api_key}",
|
|
||||||
'Content-Type': 'application/json'
|
|
||||||
}
|
|
||||||
data = {
|
|
||||||
'prompt': prompt,
|
|
||||||
'n': 1,
|
|
||||||
'size': resolution,
|
|
||||||
'model': model,
|
|
||||||
'response_format': 'url'
|
|
||||||
}
|
|
||||||
if quality is not None:
|
|
||||||
data['quality'] = quality
|
|
||||||
if style is not None:
|
|
||||||
data['style'] = style
|
|
||||||
response = requests.post(url, headers=headers, json=data, proxies=proxies)
|
|
||||||
# logger.info(response.content)
|
|
||||||
try:
|
|
||||||
image_url = json.loads(response.content.decode('utf8'))['data'][0]['url']
|
|
||||||
except:
|
|
||||||
raise RuntimeError(response.content.decode())
|
|
||||||
# 文件保存到本地
|
|
||||||
r = requests.get(image_url, proxies=proxies)
|
|
||||||
file_path = f'{get_log_folder()}/image_gen/'
|
|
||||||
os.makedirs(file_path, exist_ok=True)
|
|
||||||
file_name = 'Image' + time.strftime("%Y-%m-%d-%H-%M-%S", time.localtime()) + '.png'
|
|
||||||
with open(file_path+file_name, 'wb+') as f: f.write(r.content)
|
|
||||||
|
|
||||||
|
|
||||||
return image_url, file_path+file_name
|
|
||||||
|
|
||||||
|
|
||||||
def edit_image(llm_kwargs, prompt, image_path, resolution="1024x1024", model="dall-e-2"):
|
|
||||||
import requests, json, time, os
|
|
||||||
from request_llms.bridge_all import model_info
|
|
||||||
|
|
||||||
proxies = get_conf('proxies')
|
|
||||||
api_key = select_api_key(llm_kwargs['api_key'], llm_kwargs['llm_model'])
|
|
||||||
chat_endpoint = model_info[llm_kwargs['llm_model']]['endpoint']
|
|
||||||
# 'https://api.openai.com/v1/chat/completions'
|
|
||||||
img_endpoint = chat_endpoint.replace('chat/completions','images/edits')
|
|
||||||
# # Generate the image
|
|
||||||
url = img_endpoint
|
|
||||||
n = 1
|
|
||||||
headers = {
|
|
||||||
'Authorization': f"Bearer {api_key}",
|
|
||||||
}
|
|
||||||
make_transparent(image_path, image_path+'.tsp.png')
|
|
||||||
make_square_image(image_path+'.tsp.png', image_path+'.tspsq.png')
|
|
||||||
resize_image(image_path+'.tspsq.png', image_path+'.ready.png', max_size=1024)
|
|
||||||
image_path = image_path+'.ready.png'
|
|
||||||
with open(image_path, 'rb') as f:
|
|
||||||
file_content = f.read()
|
|
||||||
files = {
|
|
||||||
'image': (os.path.basename(image_path), file_content),
|
|
||||||
# 'mask': ('mask.png', open('mask.png', 'rb'))
|
|
||||||
'prompt': (None, prompt),
|
|
||||||
"n": (None, str(n)),
|
|
||||||
'size': (None, resolution),
|
|
||||||
}
|
|
||||||
|
|
||||||
response = requests.post(url, headers=headers, files=files, proxies=proxies)
|
|
||||||
# logger.info(response.content)
|
|
||||||
try:
|
|
||||||
image_url = json.loads(response.content.decode('utf8'))['data'][0]['url']
|
|
||||||
except:
|
|
||||||
raise RuntimeError(response.content.decode())
|
|
||||||
# 文件保存到本地
|
|
||||||
r = requests.get(image_url, proxies=proxies)
|
|
||||||
file_path = f'{get_log_folder()}/image_gen/'
|
|
||||||
os.makedirs(file_path, exist_ok=True)
|
|
||||||
file_name = 'Image' + time.strftime("%Y-%m-%d-%H-%M-%S", time.localtime()) + '.png'
|
|
||||||
with open(file_path+file_name, 'wb+') as f: f.write(r.content)
|
|
||||||
|
|
||||||
|
|
||||||
return image_url, file_path+file_name
|
|
||||||
|
|
||||||
|
|
||||||
@CatchException
|
|
||||||
def 图片生成_DALLE2(prompt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_request):
|
|
||||||
"""
|
|
||||||
txt 输入栏用户输入的文本,例如需要翻译的一段话,再例如一个包含了待处理文件的路径
|
|
||||||
llm_kwargs gpt模型参数,如温度和top_p等,一般原样传递下去就行
|
|
||||||
plugin_kwargs 插件模型的参数,暂时没有用武之地
|
|
||||||
chatbot 聊天显示框的句柄,用于显示给用户
|
|
||||||
history 聊天历史,前情提要
|
|
||||||
system_prompt 给gpt的静默提醒
|
|
||||||
user_request 当前用户的请求信息（IP地址等）
|
|
||||||
"""
|
|
||||||
history = [] # 清空历史,以免输入溢出
|
|
||||||
if prompt.strip() == "":
|
|
||||||
chatbot.append((prompt, "[Local Message] 图像生成提示为空白，请在“输入区”输入图像生成提示。"))
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面 界面更新
|
|
||||||
return
|
|
||||||
chatbot.append(("您正在调用“图像生成”插件。", "[Local Message] 生成图像, 使用前请切换模型到GPT系列。如果中文Prompt效果不理想, 请尝试英文Prompt。正在处理中 ....."))
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面 由于请求gpt需要一段时间,我们先及时地做一次界面更新
|
|
||||||
if ("advanced_arg" in plugin_kwargs) and (plugin_kwargs["advanced_arg"] == ""): plugin_kwargs.pop("advanced_arg")
|
|
||||||
resolution = plugin_kwargs.get("advanced_arg", '1024x1024')
|
|
||||||
image_url, image_path = gen_image(llm_kwargs, prompt, resolution)
|
|
||||||
chatbot.append([prompt,
|
|
||||||
f'图像中转网址: <br/>`{image_url}`<br/>'+
|
|
||||||
f'中转网址预览: <br/><div align="center"><img src="{image_url}"></div>'
|
|
||||||
f'本地文件地址: <br/>`{image_path}`<br/>'+
|
|
||||||
f'本地文件预览: <br/><div align="center"><img src="file={image_path}"></div>'
|
|
||||||
])
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面 界面更新
|
|
||||||
|
|
||||||
|
|
||||||
@CatchException
|
|
||||||
def 图片生成_DALLE3(prompt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_request):
|
|
||||||
history = [] # 清空历史,以免输入溢出
|
|
||||||
if prompt.strip() == "":
|
|
||||||
chatbot.append((prompt, "[Local Message] 图像生成提示为空白，请在“输入区”输入图像生成提示。"))
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面 界面更新
|
|
||||||
return
|
|
||||||
chatbot.append(("您正在调用“图像生成”插件。", "[Local Message] 生成图像, 使用前请切换模型到GPT系列。如果中文Prompt效果不理想, 请尝试英文Prompt。正在处理中 ....."))
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面 由于请求gpt需要一段时间,我们先及时地做一次界面更新
|
|
||||||
if ("advanced_arg" in plugin_kwargs) and (plugin_kwargs["advanced_arg"] == ""): plugin_kwargs.pop("advanced_arg")
|
|
||||||
resolution_arg = plugin_kwargs.get("advanced_arg", '1024x1024-standard-vivid').lower()
|
|
||||||
parts = resolution_arg.split('-')
|
|
||||||
resolution = parts[0] # 解析分辨率
|
|
||||||
quality = 'standard' # 质量与风格默认值
|
|
||||||
style = 'vivid'
|
|
||||||
# 遍历检查是否有额外参数
|
|
||||||
for part in parts[1:]:
|
|
||||||
if part in ['hd', 'standard']:
|
|
||||||
quality = part
|
|
||||||
elif part in ['vivid', 'natural']:
|
|
||||||
style = part
|
|
||||||
image_url, image_path = gen_image(llm_kwargs, prompt, resolution, model="dall-e-3", quality=quality, style=style)
|
|
||||||
chatbot.append([prompt,
|
|
||||||
f'图像中转网址: <br/>`{image_url}`<br/>'+
|
|
||||||
f'中转网址预览: <br/><div align="center"><img src="{image_url}"></div>'
|
|
||||||
f'本地文件地址: <br/>`{image_path}`<br/>'+
|
|
||||||
f'本地文件预览: <br/><div align="center"><img src="file={image_path}"></div>'
|
|
||||||
])
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面 界面更新
|
|
||||||
|
|
||||||
|
|
||||||
class ImageEditState(GptAcademicState):
|
|
||||||
# 尚未完成
|
|
||||||
def get_image_file(self, x):
|
|
||||||
import os, glob
|
|
||||||
if len(x) == 0: return False, None
|
|
||||||
if not os.path.exists(x): return False, None
|
|
||||||
if x.endswith('.png'): return True, x
|
|
||||||
file_manifest = [f for f in glob.glob(f'{x}/**/*.png', recursive=True)]
|
|
||||||
confirm = (len(file_manifest) >= 1 and file_manifest[0].endswith('.png') and os.path.exists(file_manifest[0]))
|
|
||||||
file = None if not confirm else file_manifest[0]
|
|
||||||
return confirm, file
|
|
||||||
|
|
||||||
def lock_plugin(self, chatbot):
|
|
||||||
chatbot._cookies['lock_plugin'] = 'crazy_functions.Image_Generate->图片修改_DALLE2'
|
|
||||||
self.dump_state(chatbot)
|
|
||||||
|
|
||||||
def unlock_plugin(self, chatbot):
|
|
||||||
self.reset()
|
|
||||||
chatbot._cookies['lock_plugin'] = None
|
|
||||||
self.dump_state(chatbot)
|
|
||||||
|
|
||||||
def get_resolution(self, x):
|
|
||||||
return (x in ['256x256', '512x512', '1024x1024']), x
|
|
||||||
|
|
||||||
def get_prompt(self, x):
|
|
||||||
confirm = (len(x)>=5) and (not self.get_resolution(x)[0]) and (not self.get_image_file(x)[0])
|
|
||||||
return confirm, x
|
|
||||||
|
|
||||||
def reset(self):
|
|
||||||
self.req = [
|
|
||||||
{'value':None, 'description': '请先上传图像（必须是.png格式）, 然后再次点击本插件', 'verify_fn': self.get_image_file},
|
|
||||||
{'value':None, 'description': '请输入分辨率,可选：256x256, 512x512 或 1024x1024, 然后再次点击本插件', 'verify_fn': self.get_resolution},
|
|
||||||
{'value':None, 'description': '请输入修改需求,建议您使用英文提示词, 然后再次点击本插件', 'verify_fn': self.get_prompt},
|
|
||||||
]
|
|
||||||
self.info = ""
|
|
||||||
|
|
||||||
def feed(self, prompt, chatbot):
|
|
||||||
for r in self.req:
|
|
||||||
if r['value'] is None:
|
|
||||||
confirm, res = r['verify_fn'](prompt)
|
|
||||||
if confirm:
|
|
||||||
r['value'] = res
|
|
||||||
self.dump_state(chatbot)
|
|
||||||
break
|
|
||||||
return self
|
|
||||||
|
|
||||||
def next_req(self):
|
|
||||||
for r in self.req:
|
|
||||||
if r['value'] is None:
|
|
||||||
return r['description']
|
|
||||||
return "已经收集到所有信息"
|
|
||||||
|
|
||||||
def already_obtained_all_materials(self):
|
|
||||||
return all([x['value'] is not None for x in self.req])
|
|
||||||
|
|
||||||
@CatchException
|
|
||||||
def 图片修改_DALLE2(prompt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_request):
|
|
||||||
# 尚未完成
|
|
||||||
history = [] # 清空历史
|
|
||||||
state = ImageEditState.get_state(chatbot, ImageEditState)
|
|
||||||
state = state.feed(prompt, chatbot)
|
|
||||||
state.lock_plugin(chatbot)
|
|
||||||
if not state.already_obtained_all_materials():
|
|
||||||
chatbot.append(["图片修改\n\n1. 上传图片（图片中需要修改的位置用橡皮擦擦除为纯白色，即RGB=255,255,255）\n2. 输入分辨率 \n3. 输入修改需求", state.next_req()])
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history)
|
|
||||||
return
|
|
||||||
|
|
||||||
image_path = state.req[0]['value']
|
|
||||||
resolution = state.req[1]['value']
|
|
||||||
prompt = state.req[2]['value']
|
|
||||||
chatbot.append(["图片修改, 执行中", f"图片:`{image_path}`<br/>分辨率:`{resolution}`<br/>修改需求:`{prompt}`"])
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history)
|
|
||||||
image_url, image_path = edit_image(llm_kwargs, prompt, image_path, resolution)
|
|
||||||
chatbot.append([prompt,
|
|
||||||
f'图像中转网址: <br/>`{image_url}`<br/>'+
|
|
||||||
f'中转网址预览: <br/><div align="center"><img src="{image_url}"></div>'
|
|
||||||
f'本地文件地址: <br/>`{image_path}`<br/>'+
|
|
||||||
f'本地文件预览: <br/><div align="center"><img src="file={image_path}"></div>'
|
|
||||||
])
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面 界面更新
|
|
||||||
state.unlock_plugin(chatbot)
|
|
||||||
|
|
||||||
def make_transparent(input_image_path, output_image_path):
|
|
||||||
from PIL import Image
|
|
||||||
image = Image.open(input_image_path)
|
|
||||||
image = image.convert("RGBA")
|
|
||||||
data = image.getdata()
|
|
||||||
new_data = []
|
|
||||||
for item in data:
|
|
||||||
if item[0] == 255 and item[1] == 255 and item[2] == 255:
|
|
||||||
new_data.append((255, 255, 255, 0))
|
|
||||||
else:
|
|
||||||
new_data.append(item)
|
|
||||||
image.putdata(new_data)
|
|
||||||
image.save(output_image_path, "PNG")
|
|
||||||
|
|
||||||
def resize_image(input_path, output_path, max_size=1024):
|
|
||||||
from PIL import Image
|
|
||||||
with Image.open(input_path) as img:
|
|
||||||
width, height = img.size
|
|
||||||
if width > max_size or height > max_size:
|
|
||||||
if width >= height:
|
|
||||||
new_width = max_size
|
|
||||||
new_height = int((max_size / width) * height)
|
|
||||||
else:
|
|
||||||
new_height = max_size
|
|
||||||
new_width = int((max_size / height) * width)
|
|
||||||
|
|
||||||
resized_img = img.resize(size=(new_width, new_height))
|
|
||||||
resized_img.save(output_path)
|
|
||||||
else:
|
|
||||||
img.save(output_path)
|
|
||||||
|
|
||||||
def make_square_image(input_path, output_path):
|
|
||||||
from PIL import Image
|
|
||||||
with Image.open(input_path) as img:
|
|
||||||
width, height = img.size
|
|
||||||
size = max(width, height)
|
|
||||||
new_img = Image.new("RGBA", (size, size), color="black")
|
|
||||||
new_img.paste(img, ((size - width) // 2, (size - height) // 2))
|
|
||||||
new_img.save(output_path)
|
|
||||||
@@ -1,311 +0,0 @@
|
|||||||
import requests
|
|
||||||
import random
|
|
||||||
import time
|
|
||||||
import re
|
|
||||||
import json
|
|
||||||
from bs4 import BeautifulSoup
|
|
||||||
from functools import lru_cache
|
|
||||||
from itertools import zip_longest
|
|
||||||
from check_proxy import check_proxy
|
|
||||||
from toolbox import CatchException, update_ui, get_conf, update_ui_lastest_msg
|
|
||||||
from crazy_functions.crazy_utils import request_gpt_model_in_new_thread_with_ui_alive, input_clipping
|
|
||||||
from request_llms.bridge_all import model_info
|
|
||||||
from request_llms.bridge_all import predict_no_ui_long_connection
|
|
||||||
from crazy_functions.prompts.internet import SearchOptimizerPrompt, SearchAcademicOptimizerPrompt
|
|
||||||
|
|
||||||
def search_optimizer(
|
|
||||||
query,
|
|
||||||
proxies,
|
|
||||||
history,
|
|
||||||
llm_kwargs,
|
|
||||||
optimizer=1,
|
|
||||||
categories="general",
|
|
||||||
searxng_url=None,
|
|
||||||
engines=None,
|
|
||||||
):
|
|
||||||
# ------------- < 第1步：尝试进行搜索优化 > -------------
|
|
||||||
# * 增强优化，会尝试结合历史记录进行搜索优化
|
|
||||||
if optimizer == 2:
|
|
||||||
his = " "
|
|
||||||
if len(history) == 0:
|
|
||||||
pass
|
|
||||||
else:
|
|
||||||
for i, h in enumerate(history):
|
|
||||||
if i % 2 == 0:
|
|
||||||
his += f"Q: {h}\n"
|
|
||||||
else:
|
|
||||||
his += f"A: {h}\n"
|
|
||||||
if categories == "general":
|
|
||||||
sys_prompt = SearchOptimizerPrompt.format(query=query, history=his, num=4)
|
|
||||||
elif categories == "science":
|
|
||||||
sys_prompt = SearchAcademicOptimizerPrompt.format(query=query, history=his, num=4)
|
|
||||||
else:
|
|
||||||
his = " "
|
|
||||||
if categories == "general":
|
|
||||||
sys_prompt = SearchOptimizerPrompt.format(query=query, history=his, num=3)
|
|
||||||
elif categories == "science":
|
|
||||||
sys_prompt = SearchAcademicOptimizerPrompt.format(query=query, history=his, num=3)
|
|
||||||
|
|
||||||
mutable = ["", time.time(), ""]
|
|
||||||
llm_kwargs["temperature"] = 0.8
|
|
||||||
try:
|
|
||||||
querys_json = predict_no_ui_long_connection(
|
|
||||||
inputs=query,
|
|
||||||
llm_kwargs=llm_kwargs,
|
|
||||||
history=[],
|
|
||||||
sys_prompt=sys_prompt,
|
|
||||||
observe_window=mutable,
|
|
||||||
)
|
|
||||||
except Exception:
|
|
||||||
querys_json = "1234"
|
|
||||||
#* 尝试解码优化后的搜索结果
|
|
||||||
querys_json = re.sub(r"```json|```", "", querys_json)
|
|
||||||
try:
|
|
||||||
querys = json.loads(querys_json)
|
|
||||||
except Exception:
|
|
||||||
#* 如果解码失败,降低温度再试一次
|
|
||||||
try:
|
|
||||||
llm_kwargs["temperature"] = 0.4
|
|
||||||
querys_json = predict_no_ui_long_connection(
|
|
||||||
inputs=query,
|
|
||||||
llm_kwargs=llm_kwargs,
|
|
||||||
history=[],
|
|
||||||
sys_prompt=sys_prompt,
|
|
||||||
observe_window=mutable,
|
|
||||||
)
|
|
||||||
querys_json = re.sub(r"```json|```", "", querys_json)
|
|
||||||
querys = json.loads(querys_json)
|
|
||||||
except Exception:
|
|
||||||
#* 如果再次失败，直接返回原始问题
|
|
||||||
querys = [query]
|
|
||||||
links = []
|
|
||||||
success = 0
|
|
||||||
Exceptions = ""
|
|
||||||
for q in querys:
|
|
||||||
try:
|
|
||||||
link = searxng_request(q, proxies, categories, searxng_url, engines=engines)
|
|
||||||
if len(link) > 0:
|
|
||||||
links.append(link[:-5])
|
|
||||||
success += 1
|
|
||||||
except Exception:
|
|
||||||
Exceptions = Exception
|
|
||||||
pass
|
|
||||||
if success == 0:
|
|
||||||
raise ValueError(f"在线搜索失败！\n{Exceptions}")
|
|
||||||
# * 清洗搜索结果，依次放入每组第一，第二个搜索结果，并清洗重复的搜索结果
|
|
||||||
seen_links = set()
|
|
||||||
result = []
|
|
||||||
for tuple in zip_longest(*links, fillvalue=None):
|
|
||||||
for item in tuple:
|
|
||||||
if item is not None:
|
|
||||||
link = item["link"]
|
|
||||||
if link not in seen_links:
|
|
||||||
seen_links.add(link)
|
|
||||||
result.append(item)
|
|
||||||
return result
|
|
||||||
|
|
||||||
|
|
||||||
@lru_cache
|
|
||||||
def get_auth_ip():
|
|
||||||
ip = check_proxy(None, return_ip=True)
|
|
||||||
if ip is None:
|
|
||||||
return '114.114.114.' + str(random.randint(1, 10))
|
|
||||||
return ip
|
|
||||||
|
|
||||||
|
|
||||||
def searxng_request(query, proxies, categories='general', searxng_url=None, engines=None):
|
|
||||||
if searxng_url is None:
|
|
||||||
urls = get_conf("SEARXNG_URLS")
|
|
||||||
url = random.choice(urls)
|
|
||||||
else:
|
|
||||||
url = searxng_url
|
|
||||||
|
|
||||||
if engines == "Mixed":
|
|
||||||
engines = None
|
|
||||||
|
|
||||||
if categories == 'general':
|
|
||||||
params = {
|
|
||||||
'q': query, # 搜索查询
|
|
||||||
'format': 'json', # 输出格式为JSON
|
|
||||||
'language': 'zh', # 搜索语言
|
|
||||||
'engines': engines,
|
|
||||||
}
|
|
||||||
elif categories == 'science':
|
|
||||||
params = {
|
|
||||||
'q': query, # 搜索查询
|
|
||||||
'format': 'json', # 输出格式为JSON
|
|
||||||
'language': 'zh', # 搜索语言
|
|
||||||
'categories': 'science'
|
|
||||||
}
|
|
||||||
else:
|
|
||||||
raise ValueError('不支持的检索类型')
|
|
||||||
|
|
||||||
headers = {
|
|
||||||
'Accept-Language': 'zh-CN,zh;q=0.9',
|
|
||||||
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36',
|
|
||||||
'X-Forwarded-For': get_auth_ip(),
|
|
||||||
'X-Real-IP': get_auth_ip()
|
|
||||||
}
|
|
||||||
results = []
|
|
||||||
response = requests.post(url, params=params, headers=headers, proxies=proxies, timeout=30)
|
|
||||||
if response.status_code == 200:
|
|
||||||
json_result = response.json()
|
|
||||||
for result in json_result['results']:
|
|
||||||
item = {
|
|
||||||
"title": result.get("title", ""),
|
|
||||||
"source": result.get("engines", "unknown"),
|
|
||||||
"content": result.get("content", ""),
|
|
||||||
"link": result["url"],
|
|
||||||
}
|
|
||||||
results.append(item)
|
|
||||||
return results
|
|
||||||
else:
|
|
||||||
if response.status_code == 429:
|
|
||||||
raise ValueError("Searxng（在线搜索服务）当前使用人数太多，请稍后。")
|
|
||||||
else:
|
|
||||||
raise ValueError("在线搜索失败，状态码: " + str(response.status_code) + '\t' + response.content.decode('utf-8'))
|
|
||||||
|
|
||||||
|
|
||||||
def scrape_text(url, proxies) -> str:
|
|
||||||
"""Scrape text from a webpage
|
|
||||||
|
|
||||||
Args:
|
|
||||||
url (str): The URL to scrape text from
|
|
||||||
|
|
||||||
Returns:
|
|
||||||
str: The scraped text
|
|
||||||
"""
|
|
||||||
headers = {
|
|
||||||
'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.61 Safari/537.36',
|
|
||||||
'Content-Type': 'text/plain',
|
|
||||||
}
|
|
||||||
try:
|
|
||||||
response = requests.get(url, headers=headers, proxies=proxies, timeout=8)
|
|
||||||
if response.encoding == "ISO-8859-1": response.encoding = response.apparent_encoding
|
|
||||||
except:
|
|
||||||
return "无法连接到该网页"
|
|
||||||
soup = BeautifulSoup(response.text, "html.parser")
|
|
||||||
for script in soup(["script", "style"]):
|
|
||||||
script.extract()
|
|
||||||
text = soup.get_text()
|
|
||||||
lines = (line.strip() for line in text.splitlines())
|
|
||||||
chunks = (phrase.strip() for line in lines for phrase in line.split(" "))
|
|
||||||
text = "\n".join(chunk for chunk in chunks if chunk)
|
|
||||||
return text
|
|
||||||
|
|
||||||
def internet_search_with_analysis_prompt(prompt, analysis_prompt, llm_kwargs, chatbot):
|
|
||||||
from toolbox import get_conf
|
|
||||||
proxies = get_conf('proxies')
|
|
||||||
categories = 'general'
|
|
||||||
searxng_url = None # 使用默认的searxng_url
|
|
||||||
engines = None # 使用默认的搜索引擎
|
|
||||||
yield from update_ui_lastest_msg(lastmsg=f"检索中: {prompt} ...", chatbot=chatbot, history=[], delay=1)
|
|
||||||
urls = searxng_request(prompt, proxies, categories, searxng_url, engines=engines)
|
|
||||||
yield from update_ui_lastest_msg(lastmsg=f"依次访问搜索到的网站 ...", chatbot=chatbot, history=[], delay=1)
|
|
||||||
if len(urls) == 0:
|
|
||||||
return None
|
|
||||||
max_search_result = 5 # 最多收纳多少个网页的结果
|
|
||||||
history = []
|
|
||||||
for index, url in enumerate(urls[:max_search_result]):
|
|
||||||
yield from update_ui_lastest_msg(lastmsg=f"依次访问搜索到的网站: {url['link']} ...", chatbot=chatbot, history=[], delay=1)
|
|
||||||
res = scrape_text(url['link'], proxies)
|
|
||||||
prefix = f"第{index}份搜索结果 [源自{url['source'][0]}搜索] （{url['title'][:25]}）："
|
|
||||||
history.extend([prefix, res])
|
|
||||||
i_say = f"从以上搜索结果中抽取信息，然后回答问题：{prompt} {analysis_prompt}"
|
|
||||||
i_say, history = input_clipping( # 裁剪输入，从最长的条目开始裁剪，防止爆token
|
|
||||||
inputs=i_say,
|
|
||||||
history=history,
|
|
||||||
max_token_limit=8192
|
|
||||||
)
|
|
||||||
gpt_say = predict_no_ui_long_connection(
|
|
||||||
inputs=i_say,
|
|
||||||
llm_kwargs=llm_kwargs,
|
|
||||||
history=history,
|
|
||||||
sys_prompt="请从搜索结果中抽取信息，对最相关的两个搜索结果进行总结，然后回答问题。",
|
|
||||||
console_slience=False,
|
|
||||||
)
|
|
||||||
return gpt_say
|
|
||||||
|
|
||||||
@CatchException
|
|
||||||
def 连接网络回答问题(txt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_request):
|
|
||||||
optimizer_history = history[:-8]
|
|
||||||
history = [] # 清空历史，以免输入溢出
|
|
||||||
chatbot.append((f"请结合互联网信息回答以下问题：{txt}", "检索中..."))
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
|
|
||||||
# ------------- < 第1步：爬取搜索引擎的结果 > -------------
|
|
||||||
from toolbox import get_conf
|
|
||||||
proxies = get_conf('proxies')
|
|
||||||
categories = plugin_kwargs.get('categories', 'general')
|
|
||||||
searxng_url = plugin_kwargs.get('searxng_url', None)
|
|
||||||
engines = plugin_kwargs.get('engine', None)
|
|
||||||
optimizer = plugin_kwargs.get('optimizer', "关闭")
|
|
||||||
if optimizer == "关闭":
|
|
||||||
urls = searxng_request(txt, proxies, categories, searxng_url, engines=engines)
|
|
||||||
else:
|
|
||||||
urls = search_optimizer(txt, proxies, optimizer_history, llm_kwargs, optimizer, categories, searxng_url, engines)
|
|
||||||
history = []
|
|
||||||
if len(urls) == 0:
|
|
||||||
chatbot.append((f"结论：{txt}",
|
|
||||||
"[Local Message] 受到限制，无法从searxng获取信息！请尝试更换搜索引擎。"))
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
|
|
||||||
# ------------- < 第2步：依次访问网页 > -------------
|
|
||||||
max_search_result = 5 # 最多收纳多少个网页的结果
|
|
||||||
if optimizer == "开启(增强)":
|
|
||||||
max_search_result = 8
|
|
||||||
chatbot.append(["联网检索中 ...", None])
|
|
||||||
for index, url in enumerate(urls[:max_search_result]):
|
|
||||||
res = scrape_text(url['link'], proxies)
|
|
||||||
prefix = f"第{index}份搜索结果 [源自{url['source'][0]}搜索] （{url['title'][:25]}）："
|
|
||||||
history.extend([prefix, res])
|
|
||||||
res_squeeze = res.replace('\n', '...')
|
|
||||||
chatbot[-1] = [prefix + "\n\n" + res_squeeze[:500] + "......", None]
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
|
|
||||||
# ------------- < 第3步：ChatGPT综合 > -------------
|
|
||||||
if (optimizer != "开启(增强)"):
|
|
||||||
i_say = f"从以上搜索结果中抽取信息，然后回答问题：{txt}"
|
|
||||||
i_say, history = input_clipping( # 裁剪输入，从最长的条目开始裁剪，防止爆token
|
|
||||||
inputs=i_say,
|
|
||||||
history=history,
|
|
||||||
max_token_limit=min(model_info[llm_kwargs['llm_model']]['max_token']*3//4, 8192)
|
|
||||||
)
|
|
||||||
gpt_say = yield from request_gpt_model_in_new_thread_with_ui_alive(
|
|
||||||
inputs=i_say, inputs_show_user=i_say,
|
|
||||||
llm_kwargs=llm_kwargs, chatbot=chatbot, history=history,
|
|
||||||
sys_prompt="请从给定的若干条搜索结果中抽取信息，对最相关的两个搜索结果进行总结，然后回答问题。"
|
|
||||||
)
|
|
||||||
chatbot[-1] = (i_say, gpt_say)
|
|
||||||
history.append(i_say);history.append(gpt_say)
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面 # 界面更新
|
|
||||||
|
|
||||||
#* 或者使用搜索优化器，这样可以保证后续问答能读取到有效的历史记录
|
|
||||||
else:
|
|
||||||
i_say = f"从以上搜索结果中抽取与问题：{txt} 相关的信息:"
|
|
||||||
i_say, history = input_clipping( # 裁剪输入，从最长的条目开始裁剪，防止爆token
|
|
||||||
inputs=i_say,
|
|
||||||
history=history,
|
|
||||||
max_token_limit=min(model_info[llm_kwargs['llm_model']]['max_token']*3//4, 8192)
|
|
||||||
)
|
|
||||||
gpt_say = yield from request_gpt_model_in_new_thread_with_ui_alive(
|
|
||||||
inputs=i_say, inputs_show_user=i_say,
|
|
||||||
llm_kwargs=llm_kwargs, chatbot=chatbot, history=history,
|
|
||||||
sys_prompt="请从给定的若干条搜索结果中抽取信息，对最相关的三个搜索结果进行总结"
|
|
||||||
)
|
|
||||||
chatbot[-1] = (i_say, gpt_say)
|
|
||||||
history = []
|
|
||||||
history.append(i_say);history.append(gpt_say)
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面 # 界面更新
|
|
||||||
|
|
||||||
# ------------- < 第4步：根据综合回答问题 > -------------
|
|
||||||
i_say = f"请根据以上搜索结果回答问题：{txt}"
|
|
||||||
gpt_say = yield from request_gpt_model_in_new_thread_with_ui_alive(
|
|
||||||
inputs=i_say, inputs_show_user=i_say,
|
|
||||||
llm_kwargs=llm_kwargs, chatbot=chatbot, history=history,
|
|
||||||
sys_prompt="请根据给定的若干条搜索结果回答问题"
|
|
||||||
)
|
|
||||||
chatbot[-1] = (i_say, gpt_say)
|
|
||||||
history.append(i_say);history.append(gpt_say)
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history)
|
|
||||||
@@ -1,48 +0,0 @@
|
|||||||
import random
|
|
||||||
from toolbox import get_conf
|
|
||||||
from crazy_functions.Internet_GPT import 连接网络回答问题
|
|
||||||
from crazy_functions.plugin_template.plugin_class_template import GptAcademicPluginTemplate, ArgProperty
|
|
||||||
|
|
||||||
|
|
||||||
class NetworkGPT_Wrap(GptAcademicPluginTemplate):
|
|
||||||
def __init__(self):
|
|
||||||
"""
|
|
||||||
请注意`execute`会执行在不同的线程中，因此您在定义和使用类变量时，应当慎之又慎！
|
|
||||||
"""
|
|
||||||
pass
|
|
||||||
|
|
||||||
def define_arg_selection_menu(self):
|
|
||||||
"""
|
|
||||||
定义插件的二级选项菜单
|
|
||||||
|
|
||||||
第一个参数，名称`main_input`，参数`type`声明这是一个文本框，文本框上方显示`title`，文本框内部显示`description`，`default_value`为默认值；
|
|
||||||
第二个参数，名称`advanced_arg`，参数`type`声明这是一个文本框，文本框上方显示`title`，文本框内部显示`description`，`default_value`为默认值；
|
|
||||||
第三个参数，名称`allow_cache`，参数`type`声明这是一个下拉菜单，下拉菜单上方显示`title`+`description`，下拉菜单的选项为`options`，`default_value`为下拉菜单默认值；
|
|
||||||
|
|
||||||
"""
|
|
||||||
urls = get_conf("SEARXNG_URLS")
|
|
||||||
url = random.choice(urls)
|
|
||||||
|
|
||||||
gui_definition = {
|
|
||||||
"main_input":
|
|
||||||
ArgProperty(title="输入问题", description="待通过互联网检索的问题，会自动读取输入框内容", default_value="", type="string").model_dump_json(), # 主输入，自动从输入框同步
|
|
||||||
"categories":
|
|
||||||
ArgProperty(title="搜索分类", options=["网页", "学术论文"], default_value="网页", description="无", type="dropdown").model_dump_json(),
|
|
||||||
"engine":
|
|
||||||
ArgProperty(title="选择搜索引擎", options=["Mixed", "bing", "google", "duckduckgo"], default_value="google", description="无", type="dropdown").model_dump_json(),
|
|
||||||
"optimizer":
|
|
||||||
ArgProperty(title="搜索优化", options=["关闭", "开启", "开启(增强)"], default_value="关闭", description="是否使用搜索增强。注意这可能会消耗较多token", type="dropdown").model_dump_json(),
|
|
||||||
"searxng_url":
|
|
||||||
ArgProperty(title="Searxng服务地址", description="输入Searxng的地址", default_value=url, type="string").model_dump_json(), # 主输入，自动从输入框同步
|
|
||||||
|
|
||||||
}
|
|
||||||
return gui_definition
|
|
||||||
|
|
||||||
def execute(txt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_request):
|
|
||||||
"""
|
|
||||||
执行插件
|
|
||||||
"""
|
|
||||||
if plugin_kwargs["categories"] == "网页": plugin_kwargs["categories"] = "general"
|
|
||||||
if plugin_kwargs["categories"] == "学术论文": plugin_kwargs["categories"] = "science"
|
|
||||||
yield from 连接网络回答问题(txt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_request)
|
|
||||||
|
|
||||||
@@ -1,595 +0,0 @@
|
|||||||
from toolbox import update_ui, trimmed_format_exc, get_conf, get_log_folder, promote_file_to_downloadzone, check_repeat_upload, map_file_to_sha256
|
|
||||||
from toolbox import CatchException, report_exception, update_ui_lastest_msg, zip_result, gen_time_str
|
|
||||||
from functools import partial
|
|
||||||
from loguru import logger
|
|
||||||
|
|
||||||
import glob, os, requests, time, json, tarfile, threading
|
|
||||||
|
|
||||||
pj = os.path.join
|
|
||||||
ARXIV_CACHE_DIR = get_conf("ARXIV_CACHE_DIR")
|
|
||||||
|
|
||||||
|
|
||||||
# =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- 工具函数 =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
|
|
||||||
# 专业词汇声明 = 'If the term "agent" is used in this section, it should be translated to "智能体". '
|
|
||||||
def switch_prompt(pfg, mode, more_requirement):
|
|
||||||
"""
|
|
||||||
Generate prompts and system prompts based on the mode for proofreading or translating.
|
|
||||||
Args:
|
|
||||||
- pfg: Proofreader or Translator instance.
|
|
||||||
- mode: A string specifying the mode, either 'proofread' or 'translate_zh'.
|
|
||||||
|
|
||||||
Returns:
|
|
||||||
- inputs_array: A list of strings containing prompts for users to respond to.
|
|
||||||
- sys_prompt_array: A list of strings containing prompts for system prompts.
|
|
||||||
"""
|
|
||||||
n_split = len(pfg.sp_file_contents)
|
|
||||||
if mode == 'proofread_en':
|
|
||||||
inputs_array = [r"Below is a section from an academic paper, proofread this section." +
|
|
||||||
r"Do not modify any latex command such as \section, \cite, \begin, \item and equations. " + more_requirement +
|
|
||||||
r"Answer me only with the revised text:" +
|
|
||||||
f"\n\n{frag}" for frag in pfg.sp_file_contents]
|
|
||||||
sys_prompt_array = ["You are a professional academic paper writer." for _ in range(n_split)]
|
|
||||||
elif mode == 'translate_zh':
|
|
||||||
inputs_array = [
|
|
||||||
r"Below is a section from an English academic paper, translate it into Chinese. " + more_requirement +
|
|
||||||
r"Do not modify any latex command such as \section, \cite, \begin, \item and equations. " +
|
|
||||||
r"Answer me only with the translated text:" +
|
|
||||||
f"\n\n{frag}" for frag in pfg.sp_file_contents]
|
|
||||||
sys_prompt_array = ["You are a professional translator." for _ in range(n_split)]
|
|
||||||
else:
|
|
||||||
assert False, "未知指令"
|
|
||||||
return inputs_array, sys_prompt_array
|
|
||||||
|
|
||||||
|
|
||||||
def desend_to_extracted_folder_if_exist(project_folder):
|
|
||||||
"""
|
|
||||||
Descend into the extracted folder if it exists, otherwise return the original folder.
|
|
||||||
|
|
||||||
Args:
|
|
||||||
- project_folder: A string specifying the folder path.
|
|
||||||
|
|
||||||
Returns:
|
|
||||||
- A string specifying the path to the extracted folder, or the original folder if there is no extracted folder.
|
|
||||||
"""
|
|
||||||
maybe_dir = [f for f in glob.glob(f'{project_folder}/*') if os.path.isdir(f)]
|
|
||||||
if len(maybe_dir) == 0: return project_folder
|
|
||||||
if maybe_dir[0].endswith('.extract'): return maybe_dir[0]
|
|
||||||
return project_folder
|
|
||||||
|
|
||||||
|
|
||||||
def move_project(project_folder, arxiv_id=None):
|
|
||||||
"""
|
|
||||||
Create a new work folder and copy the project folder to it.
|
|
||||||
|
|
||||||
Args:
|
|
||||||
- project_folder: A string specifying the folder path of the project.
|
|
||||||
|
|
||||||
Returns:
|
|
||||||
- A string specifying the path to the new work folder.
|
|
||||||
"""
|
|
||||||
import shutil, time
|
|
||||||
time.sleep(2) # avoid time string conflict
|
|
||||||
if arxiv_id is not None:
|
|
||||||
new_workfolder = pj(ARXIV_CACHE_DIR, arxiv_id, 'workfolder')
|
|
||||||
else:
|
|
||||||
new_workfolder = f'{get_log_folder()}/{gen_time_str()}'
|
|
||||||
try:
|
|
||||||
shutil.rmtree(new_workfolder)
|
|
||||||
except:
|
|
||||||
pass
|
|
||||||
|
|
||||||
# align subfolder if there is a folder wrapper
|
|
||||||
items = glob.glob(pj(project_folder, '*'))
|
|
||||||
items = [item for item in items if os.path.basename(item) != '__MACOSX']
|
|
||||||
if len(glob.glob(pj(project_folder, '*.tex'))) == 0 and len(items) == 1:
|
|
||||||
if os.path.isdir(items[0]): project_folder = items[0]
|
|
||||||
|
|
||||||
shutil.copytree(src=project_folder, dst=new_workfolder)
|
|
||||||
return new_workfolder
|
|
||||||
|
|
||||||
|
|
||||||
def arxiv_download(chatbot, history, txt, allow_cache=True):
|
|
||||||
def check_cached_translation_pdf(arxiv_id):
|
|
||||||
translation_dir = pj(ARXIV_CACHE_DIR, arxiv_id, 'translation')
|
|
||||||
if not os.path.exists(translation_dir):
|
|
||||||
os.makedirs(translation_dir)
|
|
||||||
target_file = pj(translation_dir, 'translate_zh.pdf')
|
|
||||||
if os.path.exists(target_file):
|
|
||||||
promote_file_to_downloadzone(target_file, rename_file=None, chatbot=chatbot)
|
|
||||||
target_file_compare = pj(translation_dir, 'comparison.pdf')
|
|
||||||
if os.path.exists(target_file_compare):
|
|
||||||
promote_file_to_downloadzone(target_file_compare, rename_file=None, chatbot=chatbot)
|
|
||||||
return target_file
|
|
||||||
return False
|
|
||||||
|
|
||||||
def is_float(s):
|
|
||||||
try:
|
|
||||||
float(s)
|
|
||||||
return True
|
|
||||||
except ValueError:
|
|
||||||
return False
|
|
||||||
|
|
||||||
if txt.startswith('https://arxiv.org/pdf/'):
|
|
||||||
arxiv_id = txt.split('/')[-1] # 2402.14207v2.pdf
|
|
||||||
txt = arxiv_id.split('v')[0] # 2402.14207
|
|
||||||
|
|
||||||
if ('.' in txt) and ('/' not in txt) and is_float(txt): # is arxiv ID
|
|
||||||
txt = 'https://arxiv.org/abs/' + txt.strip()
|
|
||||||
if ('.' in txt) and ('/' not in txt) and is_float(txt[:10]): # is arxiv ID
|
|
||||||
txt = 'https://arxiv.org/abs/' + txt[:10]
|
|
||||||
|
|
||||||
if not txt.startswith('https://arxiv.org'):
|
|
||||||
return txt, None # 是本地文件，跳过下载
|
|
||||||
|
|
||||||
# <-------------- inspect format ------------->
|
|
||||||
chatbot.append([f"检测到arxiv文档连接", '尝试下载 ...'])
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history)
|
|
||||||
time.sleep(1) # 刷新界面
|
|
||||||
|
|
||||||
url_ = txt # https://arxiv.org/abs/1707.06690
|
|
||||||
|
|
||||||
if not txt.startswith('https://arxiv.org/abs/'):
|
|
||||||
msg = f"解析arxiv网址失败, 期望格式例如: https://arxiv.org/abs/1707.06690。实际得到格式: {url_}。"
|
|
||||||
yield from update_ui_lastest_msg(msg, chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return msg, None
|
|
||||||
# <-------------- set format ------------->
|
|
||||||
arxiv_id = url_.split('/abs/')[-1]
|
|
||||||
if 'v' in arxiv_id: arxiv_id = arxiv_id[:10]
|
|
||||||
cached_translation_pdf = check_cached_translation_pdf(arxiv_id)
|
|
||||||
if cached_translation_pdf and allow_cache: return cached_translation_pdf, arxiv_id
|
|
||||||
|
|
||||||
extract_dst = pj(ARXIV_CACHE_DIR, arxiv_id, 'extract')
|
|
||||||
translation_dir = pj(ARXIV_CACHE_DIR, arxiv_id, 'e-print')
|
|
||||||
dst = pj(translation_dir, arxiv_id + '.tar')
|
|
||||||
os.makedirs(translation_dir, exist_ok=True)
|
|
||||||
# <-------------- download arxiv source file ------------->
|
|
||||||
|
|
||||||
def fix_url_and_download():
|
|
||||||
# for url_tar in [url_.replace('/abs/', '/e-print/'), url_.replace('/abs/', '/src/')]:
|
|
||||||
for url_tar in [url_.replace('/abs/', '/src/'), url_.replace('/abs/', '/e-print/')]:
|
|
||||||
proxies = get_conf('proxies')
|
|
||||||
r = requests.get(url_tar, proxies=proxies)
|
|
||||||
if r.status_code == 200:
|
|
||||||
with open(dst, 'wb+') as f:
|
|
||||||
f.write(r.content)
|
|
||||||
return True
|
|
||||||
return False
|
|
||||||
|
|
||||||
if os.path.exists(dst) and allow_cache:
|
|
||||||
yield from update_ui_lastest_msg(f"调用缓存 {arxiv_id}", chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
success = True
|
|
||||||
else:
|
|
||||||
yield from update_ui_lastest_msg(f"开始下载 {arxiv_id}", chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
success = fix_url_and_download()
|
|
||||||
yield from update_ui_lastest_msg(f"下载完成 {arxiv_id}", chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
|
|
||||||
|
|
||||||
if not success:
|
|
||||||
yield from update_ui_lastest_msg(f"下载失败 {arxiv_id}", chatbot=chatbot, history=history)
|
|
||||||
raise tarfile.ReadError(f"论文下载失败 {arxiv_id}")
|
|
||||||
|
|
||||||
# <-------------- extract file ------------->
|
|
||||||
from toolbox import extract_archive
|
|
||||||
try:
|
|
||||||
extract_archive(file_path=dst, dest_dir=extract_dst)
|
|
||||||
except tarfile.ReadError:
|
|
||||||
os.remove(dst)
|
|
||||||
raise tarfile.ReadError(f"论文下载失败")
|
|
||||||
return extract_dst, arxiv_id
|
|
||||||
|
|
||||||
|
|
||||||
def pdf2tex_project(pdf_file_path, plugin_kwargs):
|
|
||||||
if plugin_kwargs["method"] == "MATHPIX":
|
|
||||||
# Mathpix API credentials
|
|
||||||
app_id, app_key = get_conf('MATHPIX_APPID', 'MATHPIX_APPKEY')
|
|
||||||
headers = {"app_id": app_id, "app_key": app_key}
|
|
||||||
|
|
||||||
# Step 1: Send PDF file for processing
|
|
||||||
options = {
|
|
||||||
"conversion_formats": {"tex.zip": True},
|
|
||||||
"math_inline_delimiters": ["$", "$"],
|
|
||||||
"rm_spaces": True
|
|
||||||
}
|
|
||||||
|
|
||||||
response = requests.post(url="https://api.mathpix.com/v3/pdf",
|
|
||||||
headers=headers,
|
|
||||||
data={"options_json": json.dumps(options)},
|
|
||||||
files={"file": open(pdf_file_path, "rb")})
|
|
||||||
|
|
||||||
if response.ok:
|
|
||||||
pdf_id = response.json()["pdf_id"]
|
|
||||||
logger.info(f"PDF processing initiated. PDF ID: {pdf_id}")
|
|
||||||
|
|
||||||
# Step 2: Check processing status
|
|
||||||
while True:
|
|
||||||
conversion_response = requests.get(f"https://api.mathpix.com/v3/pdf/{pdf_id}", headers=headers)
|
|
||||||
conversion_data = conversion_response.json()
|
|
||||||
|
|
||||||
if conversion_data["status"] == "completed":
|
|
||||||
logger.info("PDF processing completed.")
|
|
||||||
break
|
|
||||||
elif conversion_data["status"] == "error":
|
|
||||||
logger.info("Error occurred during processing.")
|
|
||||||
else:
|
|
||||||
logger.info(f"Processing status: {conversion_data['status']}")
|
|
||||||
time.sleep(5) # wait for a few seconds before checking again
|
|
||||||
|
|
||||||
# Step 3: Save results to local files
|
|
||||||
output_dir = os.path.join(os.path.dirname(pdf_file_path), 'mathpix_output')
|
|
||||||
if not os.path.exists(output_dir):
|
|
||||||
os.makedirs(output_dir)
|
|
||||||
|
|
||||||
url = f"https://api.mathpix.com/v3/pdf/{pdf_id}.tex"
|
|
||||||
response = requests.get(url, headers=headers)
|
|
||||||
file_name_wo_dot = '_'.join(os.path.basename(pdf_file_path).split('.')[:-1])
|
|
||||||
output_name = f"{file_name_wo_dot}.tex.zip"
|
|
||||||
output_path = os.path.join(output_dir, output_name)
|
|
||||||
with open(output_path, "wb") as output_file:
|
|
||||||
output_file.write(response.content)
|
|
||||||
logger.info(f"tex.zip file saved at: {output_path}")
|
|
||||||
|
|
||||||
import zipfile
|
|
||||||
unzip_dir = os.path.join(output_dir, file_name_wo_dot)
|
|
||||||
with zipfile.ZipFile(output_path, 'r') as zip_ref:
|
|
||||||
zip_ref.extractall(unzip_dir)
|
|
||||||
|
|
||||||
return unzip_dir
|
|
||||||
|
|
||||||
else:
|
|
||||||
logger.error(f"Error sending PDF for processing. Status code: {response.status_code}")
|
|
||||||
return None
|
|
||||||
else:
|
|
||||||
from crazy_functions.pdf_fns.parse_pdf_via_doc2x import 解析PDF_DOC2X_转Latex
|
|
||||||
unzip_dir = 解析PDF_DOC2X_转Latex(pdf_file_path)
|
|
||||||
return unzip_dir
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
# =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= 插件主程序1 =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
|
|
||||||
|
|
||||||
|
|
||||||
@CatchException
|
|
||||||
def Latex英文纠错加PDF对比(txt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_request):
|
|
||||||
# <-------------- information about this plugin ------------->
|
|
||||||
chatbot.append(["函数插件功能？",
|
|
||||||
"对整个Latex项目进行纠错, 用latex编译为PDF对修正处做高亮。函数插件贡献者: Binary-Husky。注意事项: 目前对机器学习类文献转化效果最好，其他类型文献转化效果未知。仅在Windows系统进行了测试，其他操作系统表现未知。"])
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
|
|
||||||
# <-------------- more requirements ------------->
|
|
||||||
if ("advanced_arg" in plugin_kwargs) and (plugin_kwargs["advanced_arg"] == ""): plugin_kwargs.pop("advanced_arg")
|
|
||||||
more_req = plugin_kwargs.get("advanced_arg", "")
|
|
||||||
_switch_prompt_ = partial(switch_prompt, more_requirement=more_req)
|
|
||||||
|
|
||||||
# <-------------- check deps ------------->
|
|
||||||
try:
|
|
||||||
import glob, os, time, subprocess
|
|
||||||
subprocess.Popen(['pdflatex', '-version'])
|
|
||||||
from .latex_fns.latex_actions import Latex精细分解与转化, 编译Latex
|
|
||||||
except Exception as e:
|
|
||||||
chatbot.append([f"解析项目: {txt}",
|
|
||||||
f"尝试执行Latex指令失败。Latex没有安装, 或者不在环境变量PATH中。安装方法https://tug.org/texlive/。报错信息\n\n```\n\n{trimmed_format_exc()}\n\n```\n\n"])
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
|
|
||||||
# <-------------- clear history and read input ------------->
|
|
||||||
history = []
|
|
||||||
if os.path.exists(txt):
|
|
||||||
project_folder = txt
|
|
||||||
else:
|
|
||||||
if txt == "": txt = '空空如也的输入栏'
|
|
||||||
report_exception(chatbot, history, a=f"解析项目: {txt}", b=f"找不到本地项目或无权访问: {txt}")
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
file_manifest = [f for f in glob.glob(f'{project_folder}/**/*.tex', recursive=True)]
|
|
||||||
if len(file_manifest) == 0:
|
|
||||||
report_exception(chatbot, history, a=f"解析项目: {txt}", b=f"找不到任何.tex文件: {txt}")
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
|
|
||||||
# <-------------- if is a zip/tar file ------------->
|
|
||||||
project_folder = desend_to_extracted_folder_if_exist(project_folder)
|
|
||||||
|
|
||||||
# <-------------- move latex project away from temp folder ------------->
|
|
||||||
from shared_utils.fastapi_server import validate_path_safety
|
|
||||||
validate_path_safety(project_folder, chatbot.get_user())
|
|
||||||
project_folder = move_project(project_folder, arxiv_id=None)
|
|
||||||
|
|
||||||
# <-------------- if merge_translate_zh is already generated, skip gpt req ------------->
|
|
||||||
if not os.path.exists(project_folder + '/merge_proofread_en.tex'):
|
|
||||||
yield from Latex精细分解与转化(file_manifest, project_folder, llm_kwargs, plugin_kwargs,
|
|
||||||
chatbot, history, system_prompt, mode='proofread_en',
|
|
||||||
switch_prompt=_switch_prompt_)
|
|
||||||
|
|
||||||
# <-------------- compile PDF ------------->
|
|
||||||
success = yield from 编译Latex(chatbot, history, main_file_original='merge',
|
|
||||||
main_file_modified='merge_proofread_en',
|
|
||||||
work_folder_original=project_folder, work_folder_modified=project_folder,
|
|
||||||
work_folder=project_folder)
|
|
||||||
|
|
||||||
# <-------------- zip PDF ------------->
|
|
||||||
zip_res = zip_result(project_folder)
|
|
||||||
if success:
|
|
||||||
chatbot.append((f"成功啦", '请查收结果（压缩包）...'))
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history);
|
|
||||||
time.sleep(1) # 刷新界面
|
|
||||||
promote_file_to_downloadzone(file=zip_res, chatbot=chatbot)
|
|
||||||
else:
|
|
||||||
chatbot.append((f"失败了",
|
|
||||||
'虽然PDF生成失败了, 但请查收结果（压缩包）, 内含已经翻译的Tex文档, 也是可读的, 您可以到Github Issue区, 用该压缩包+Conversation_To_File进行反馈 ...'))
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history);
|
|
||||||
time.sleep(1) # 刷新界面
|
|
||||||
promote_file_to_downloadzone(file=zip_res, chatbot=chatbot)
|
|
||||||
|
|
||||||
# <-------------- we are done ------------->
|
|
||||||
return success
|
|
||||||
|
|
||||||
|
|
||||||
# =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= 插件主程序2 =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
|
|
||||||
|
|
||||||
@CatchException
|
|
||||||
def Latex翻译中文并重新编译PDF(txt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_request):
|
|
||||||
# <-------------- information about this plugin ------------->
|
|
||||||
chatbot.append([
|
|
||||||
"函数插件功能？",
|
|
||||||
"对整个Latex项目进行翻译, 生成中文PDF。函数插件贡献者: Binary-Husky。注意事项: 此插件Windows支持最佳，Linux下必须使用Docker安装，详见项目主README.md。目前对机器学习类文献转化效果最好，其他类型文献转化效果未知。"])
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
|
|
||||||
# <-------------- more requirements ------------->
|
|
||||||
if ("advanced_arg" in plugin_kwargs) and (plugin_kwargs["advanced_arg"] == ""): plugin_kwargs.pop("advanced_arg")
|
|
||||||
more_req = plugin_kwargs.get("advanced_arg", "")
|
|
||||||
|
|
||||||
no_cache = ("--no-cache" in more_req)
|
|
||||||
if no_cache: more_req = more_req.replace("--no-cache", "").strip()
|
|
||||||
|
|
||||||
allow_gptac_cloud_io = ("--allow-cloudio" in more_req) # 从云端下载翻译结果，以及上传翻译结果到云端
|
|
||||||
if allow_gptac_cloud_io: more_req = more_req.replace("--allow-cloudio", "").strip()
|
|
||||||
|
|
||||||
allow_cache = not no_cache
|
|
||||||
_switch_prompt_ = partial(switch_prompt, more_requirement=more_req)
|
|
||||||
|
|
||||||
|
|
||||||
# <-------------- check deps ------------->
|
|
||||||
try:
|
|
||||||
import glob, os, time, subprocess
|
|
||||||
subprocess.Popen(['pdflatex', '-version'])
|
|
||||||
from .latex_fns.latex_actions import Latex精细分解与转化, 编译Latex
|
|
||||||
except Exception as e:
|
|
||||||
chatbot.append([f"解析项目: {txt}",
|
|
||||||
f"尝试执行Latex指令失败。Latex没有安装, 或者不在环境变量PATH中。安装方法https://tug.org/texlive/。报错信息\n\n```\n\n{trimmed_format_exc()}\n\n```\n\n"])
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
|
|
||||||
# <-------------- clear history and read input ------------->
|
|
||||||
history = []
|
|
||||||
try:
|
|
||||||
txt, arxiv_id = yield from arxiv_download(chatbot, history, txt, allow_cache)
|
|
||||||
except tarfile.ReadError as e:
|
|
||||||
yield from update_ui_lastest_msg(
|
|
||||||
"无法自动下载该论文的Latex源码，请前往arxiv打开此论文下载页面，点other Formats，然后download source手动下载latex源码包。接下来调用本地Latex翻译插件即可。",
|
|
||||||
chatbot=chatbot, history=history)
|
|
||||||
return
|
|
||||||
|
|
||||||
if txt.endswith('.pdf'):
|
|
||||||
report_exception(chatbot, history, a=f"解析项目: {txt}", b=f"发现已经存在翻译好的PDF文档")
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
|
|
||||||
# #################################################################
|
|
||||||
if allow_gptac_cloud_io and arxiv_id:
|
|
||||||
# 访问 GPTAC学术云，查询云端是否存在该论文的翻译版本
|
|
||||||
from crazy_functions.latex_fns.latex_actions import check_gptac_cloud
|
|
||||||
success, downloaded = check_gptac_cloud(arxiv_id, chatbot)
|
|
||||||
if success:
|
|
||||||
chatbot.append([
|
|
||||||
f"检测到GPTAC云端存在翻译版本, 如果不满意翻译结果, 请禁用云端分享, 然后重新执行。",
|
|
||||||
None
|
|
||||||
])
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history)
|
|
||||||
return
|
|
||||||
#################################################################
|
|
||||||
|
|
||||||
if os.path.exists(txt):
|
|
||||||
project_folder = txt
|
|
||||||
else:
|
|
||||||
if txt == "": txt = '空空如也的输入栏'
|
|
||||||
report_exception(chatbot, history, a=f"解析项目: {txt}", b=f"找不到本地项目或无法处理: {txt}")
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
|
|
||||||
file_manifest = [f for f in glob.glob(f'{project_folder}/**/*.tex', recursive=True)]
|
|
||||||
if len(file_manifest) == 0:
|
|
||||||
report_exception(chatbot, history, a=f"解析项目: {txt}", b=f"找不到任何.tex文件: {txt}")
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
|
|
||||||
# <-------------- if is a zip/tar file ------------->
|
|
||||||
project_folder = desend_to_extracted_folder_if_exist(project_folder)
|
|
||||||
|
|
||||||
# <-------------- move latex project away from temp folder ------------->
|
|
||||||
from shared_utils.fastapi_server import validate_path_safety
|
|
||||||
validate_path_safety(project_folder, chatbot.get_user())
|
|
||||||
project_folder = move_project(project_folder, arxiv_id)
|
|
||||||
|
|
||||||
# <-------------- if merge_translate_zh is already generated, skip gpt req ------------->
|
|
||||||
if not os.path.exists(project_folder + '/merge_translate_zh.tex'):
|
|
||||||
yield from Latex精细分解与转化(file_manifest, project_folder, llm_kwargs, plugin_kwargs,
|
|
||||||
chatbot, history, system_prompt, mode='translate_zh',
|
|
||||||
switch_prompt=_switch_prompt_)
|
|
||||||
|
|
||||||
# <-------------- compile PDF ------------->
|
|
||||||
success = yield from 编译Latex(chatbot, history, main_file_original='merge',
|
|
||||||
main_file_modified='merge_translate_zh', mode='translate_zh',
|
|
||||||
work_folder_original=project_folder, work_folder_modified=project_folder,
|
|
||||||
work_folder=project_folder)
|
|
||||||
|
|
||||||
# <-------------- zip PDF ------------->
|
|
||||||
zip_res = zip_result(project_folder)
|
|
||||||
if success:
|
|
||||||
if allow_gptac_cloud_io and arxiv_id:
|
|
||||||
# 如果用户允许，我们将翻译好的arxiv论文PDF上传到GPTAC学术云
|
|
||||||
from crazy_functions.latex_fns.latex_actions import upload_to_gptac_cloud_if_user_allow
|
|
||||||
threading.Thread(target=upload_to_gptac_cloud_if_user_allow,
|
|
||||||
args=(chatbot, arxiv_id), daemon=True).start()
|
|
||||||
|
|
||||||
chatbot.append((f"成功啦", '请查收结果（压缩包）...'))
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history)
|
|
||||||
time.sleep(1) # 刷新界面
|
|
||||||
promote_file_to_downloadzone(file=zip_res, chatbot=chatbot)
|
|
||||||
|
|
||||||
else:
|
|
||||||
chatbot.append((f"失败了",
|
|
||||||
'虽然PDF生成失败了, 但请查收结果（压缩包）, 内含已经翻译的Tex文档, 您可以到Github Issue区, 用该压缩包进行反馈。如系统是Linux，请检查系统字体（见Github wiki） ...'))
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history)
|
|
||||||
time.sleep(1) # 刷新界面
|
|
||||||
promote_file_to_downloadzone(file=zip_res, chatbot=chatbot)
|
|
||||||
|
|
||||||
# <-------------- we are done ------------->
|
|
||||||
return success
|
|
||||||
|
|
||||||
|
|
||||||
# =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- 插件主程序3 =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
|
|
||||||
|
|
||||||
@CatchException
|
|
||||||
def PDF翻译中文并重新编译PDF(txt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, web_port):
|
|
||||||
# <-------------- information about this plugin ------------->
|
|
||||||
chatbot.append([
|
|
||||||
"函数插件功能？",
|
|
||||||
"将PDF转换为Latex项目，翻译为中文后重新编译为PDF。函数插件贡献者: Marroh。注意事项: 此插件Windows支持最佳，Linux下必须使用Docker安装，详见项目主README.md。目前对机器学习类文献转化效果最好，其他类型文献转化效果未知。"])
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
|
|
||||||
# <-------------- more requirements ------------->
|
|
||||||
if ("advanced_arg" in plugin_kwargs) and (plugin_kwargs["advanced_arg"] == ""): plugin_kwargs.pop("advanced_arg")
|
|
||||||
more_req = plugin_kwargs.get("advanced_arg", "")
|
|
||||||
no_cache = more_req.startswith("--no-cache")
|
|
||||||
if no_cache: more_req.lstrip("--no-cache")
|
|
||||||
allow_cache = not no_cache
|
|
||||||
_switch_prompt_ = partial(switch_prompt, more_requirement=more_req)
|
|
||||||
|
|
||||||
# <-------------- check deps ------------->
|
|
||||||
try:
|
|
||||||
import glob, os, time, subprocess
|
|
||||||
subprocess.Popen(['pdflatex', '-version'])
|
|
||||||
from .latex_fns.latex_actions import Latex精细分解与转化, 编译Latex
|
|
||||||
except Exception as e:
|
|
||||||
chatbot.append([f"解析项目: {txt}",
|
|
||||||
f"尝试执行Latex指令失败。Latex没有安装, 或者不在环境变量PATH中。安装方法https://tug.org/texlive/。报错信息\n\n```\n\n{trimmed_format_exc()}\n\n```\n\n"])
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
|
|
||||||
# <-------------- clear history and read input ------------->
|
|
||||||
if os.path.exists(txt):
|
|
||||||
project_folder = txt
|
|
||||||
else:
|
|
||||||
if txt == "": txt = '空空如也的输入栏'
|
|
||||||
report_exception(chatbot, history, a=f"解析项目: {txt}", b=f"找不到本地项目或无法处理: {txt}")
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
|
|
||||||
file_manifest = [f for f in glob.glob(f'{project_folder}/**/*.pdf', recursive=True)]
|
|
||||||
if len(file_manifest) == 0:
|
|
||||||
report_exception(chatbot, history, a=f"解析项目: {txt}", b=f"找不到任何.pdf文件: {txt}")
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
if len(file_manifest) != 1:
|
|
||||||
report_exception(chatbot, history, a=f"解析项目: {txt}", b=f"不支持同时处理多个pdf文件: {txt}")
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
|
|
||||||
if plugin_kwargs.get("method", "") == 'MATHPIX':
|
|
||||||
app_id, app_key = get_conf('MATHPIX_APPID', 'MATHPIX_APPKEY')
|
|
||||||
if len(app_id) == 0 or len(app_key) == 0:
|
|
||||||
report_exception(chatbot, history, a="缺失 MATHPIX_APPID 和 MATHPIX_APPKEY。", b=f"请配置 MATHPIX_APPID 和 MATHPIX_APPKEY")
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
if plugin_kwargs.get("method", "") == 'DOC2X':
|
|
||||||
app_id, app_key = "", ""
|
|
||||||
DOC2X_API_KEY = get_conf('DOC2X_API_KEY')
|
|
||||||
if len(DOC2X_API_KEY) == 0:
|
|
||||||
report_exception(chatbot, history, a="缺失 DOC2X_API_KEY。", b=f"请配置 DOC2X_API_KEY")
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
|
|
||||||
hash_tag = map_file_to_sha256(file_manifest[0])
|
|
||||||
|
|
||||||
# # <-------------- check repeated pdf ------------->
|
|
||||||
# chatbot.append([f"检查PDF是否被重复上传", "正在检查..."])
|
|
||||||
# yield from update_ui(chatbot=chatbot, history=history)
|
|
||||||
# repeat, project_folder = check_repeat_upload(file_manifest[0], hash_tag)
|
|
||||||
|
|
||||||
# if repeat:
|
|
||||||
# yield from update_ui_lastest_msg(f"发现重复上传，请查收结果（压缩包）...", chatbot=chatbot, history=history)
|
|
||||||
# try:
|
|
||||||
# translate_pdf = [f for f in glob.glob(f'{project_folder}/**/merge_translate_zh.pdf', recursive=True)][0]
|
|
||||||
# promote_file_to_downloadzone(translate_pdf, rename_file=None, chatbot=chatbot)
|
|
||||||
# comparison_pdf = [f for f in glob.glob(f'{project_folder}/**/comparison.pdf', recursive=True)][0]
|
|
||||||
# promote_file_to_downloadzone(comparison_pdf, rename_file=None, chatbot=chatbot)
|
|
||||||
# zip_res = zip_result(project_folder)
|
|
||||||
# promote_file_to_downloadzone(file=zip_res, chatbot=chatbot)
|
|
||||||
# return
|
|
||||||
# except:
|
|
||||||
# report_exception(chatbot, history, a=f"解析项目: {txt}", b=f"发现重复上传，但是无法找到相关文件")
|
|
||||||
# yield from update_ui(chatbot=chatbot, history=history)
|
|
||||||
# else:
|
|
||||||
# yield from update_ui_lastest_msg(f"未发现重复上传", chatbot=chatbot, history=history)
|
|
||||||
|
|
||||||
# <-------------- convert pdf into tex ------------->
|
|
||||||
chatbot.append([f"解析项目: {txt}", "正在将PDF转换为tex项目，请耐心等待..."])
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history)
|
|
||||||
project_folder = pdf2tex_project(file_manifest[0], plugin_kwargs)
|
|
||||||
if project_folder is None:
|
|
||||||
report_exception(chatbot, history, a=f"解析项目: {txt}", b=f"PDF转换为tex项目失败")
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history)
|
|
||||||
return False
|
|
||||||
|
|
||||||
# <-------------- translate latex file into Chinese ------------->
|
|
||||||
yield from update_ui_lastest_msg("正在tex项目将翻译为中文...", chatbot=chatbot, history=history)
|
|
||||||
file_manifest = [f for f in glob.glob(f'{project_folder}/**/*.tex', recursive=True)]
|
|
||||||
if len(file_manifest) == 0:
|
|
||||||
report_exception(chatbot, history, a=f"解析项目: {txt}", b=f"找不到任何.tex文件: {txt}")
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
|
|
||||||
# <-------------- if is a zip/tar file ------------->
|
|
||||||
project_folder = desend_to_extracted_folder_if_exist(project_folder)
|
|
||||||
|
|
||||||
# <-------------- move latex project away from temp folder ------------->
|
|
||||||
from shared_utils.fastapi_server import validate_path_safety
|
|
||||||
validate_path_safety(project_folder, chatbot.get_user())
|
|
||||||
project_folder = move_project(project_folder)
|
|
||||||
|
|
||||||
# <-------------- set a hash tag for repeat-checking ------------->
|
|
||||||
with open(pj(project_folder, hash_tag + '.tag'), 'w', encoding='utf8') as f:
|
|
||||||
f.write(hash_tag)
|
|
||||||
f.close()
|
|
||||||
|
|
||||||
|
|
||||||
# <-------------- if merge_translate_zh is already generated, skip gpt req ------------->
|
|
||||||
if not os.path.exists(project_folder + '/merge_translate_zh.tex'):
|
|
||||||
yield from Latex精细分解与转化(file_manifest, project_folder, llm_kwargs, plugin_kwargs,
|
|
||||||
chatbot, history, system_prompt, mode='translate_zh',
|
|
||||||
switch_prompt=_switch_prompt_)
|
|
||||||
|
|
||||||
# <-------------- compile PDF ------------->
|
|
||||||
yield from update_ui_lastest_msg("正在将翻译好的项目tex项目编译为PDF...", chatbot=chatbot, history=history)
|
|
||||||
success = yield from 编译Latex(chatbot, history, main_file_original='merge',
|
|
||||||
main_file_modified='merge_translate_zh', mode='translate_zh',
|
|
||||||
work_folder_original=project_folder, work_folder_modified=project_folder,
|
|
||||||
work_folder=project_folder)
|
|
||||||
|
|
||||||
# <-------------- zip PDF ------------->
|
|
||||||
zip_res = zip_result(project_folder)
|
|
||||||
if success:
|
|
||||||
chatbot.append((f"成功啦", '请查收结果（压缩包）...'))
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history);
|
|
||||||
time.sleep(1) # 刷新界面
|
|
||||||
promote_file_to_downloadzone(file=zip_res, chatbot=chatbot)
|
|
||||||
else:
|
|
||||||
chatbot.append((f"失败了",
|
|
||||||
'虽然PDF生成失败了, 但请查收结果（压缩包）, 内含已经翻译的Tex文档, 您可以到Github Issue区, 用该压缩包进行反馈。如系统是Linux，请检查系统字体（见Github wiki） ...'))
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history);
|
|
||||||
time.sleep(1) # 刷新界面
|
|
||||||
promote_file_to_downloadzone(file=zip_res, chatbot=chatbot)
|
|
||||||
|
|
||||||
# <-------------- we are done ------------->
|
|
||||||
return success
|
|
||||||
@@ -1,85 +0,0 @@
|
|||||||
|
|
||||||
from crazy_functions.Latex_Function import Latex翻译中文并重新编译PDF, PDF翻译中文并重新编译PDF
|
|
||||||
from crazy_functions.plugin_template.plugin_class_template import GptAcademicPluginTemplate, ArgProperty
|
|
||||||
|
|
||||||
|
|
||||||
class Arxiv_Localize(GptAcademicPluginTemplate):
|
|
||||||
def __init__(self):
|
|
||||||
"""
|
|
||||||
请注意`execute`会执行在不同的线程中，因此您在定义和使用类变量时，应当慎之又慎！
|
|
||||||
"""
|
|
||||||
pass
|
|
||||||
|
|
||||||
def define_arg_selection_menu(self):
|
|
||||||
"""
|
|
||||||
定义插件的二级选项菜单
|
|
||||||
|
|
||||||
第一个参数，名称`main_input`，参数`type`声明这是一个文本框，文本框上方显示`title`，文本框内部显示`description`，`default_value`为默认值；
|
|
||||||
第二个参数，名称`advanced_arg`，参数`type`声明这是一个文本框，文本框上方显示`title`，文本框内部显示`description`，`default_value`为默认值；
|
|
||||||
第三个参数，名称`allow_cache`，参数`type`声明这是一个下拉菜单，下拉菜单上方显示`title`+`description`，下拉菜单的选项为`options`，`default_value`为下拉菜单默认值；
|
|
||||||
|
|
||||||
"""
|
|
||||||
gui_definition = {
|
|
||||||
"main_input":
|
|
||||||
ArgProperty(title="ArxivID", description="输入Arxiv的ID或者网址", default_value="", type="string").model_dump_json(), # 主输入，自动从输入框同步
|
|
||||||
"advanced_arg":
|
|
||||||
ArgProperty(title="额外的翻译提示词",
|
|
||||||
description=r"如果有必要, 请在此处给出自定义翻译命令, 解决部分词汇翻译不准确的问题。 "
|
|
||||||
r"例如当单词'agent'翻译不准确时, 请尝试把以下指令复制到高级参数区: "
|
|
||||||
r'If the term "agent" is used in this section, it should be translated to "智能体". ',
|
|
||||||
default_value="", type="string").model_dump_json(), # 高级参数输入区，自动同步
|
|
||||||
"allow_cache":
|
|
||||||
ArgProperty(title="是否允许从缓存中调取结果", options=["允许缓存", "从头执行"], default_value="允许缓存", description="无", type="dropdown").model_dump_json(),
|
|
||||||
"allow_cloudio":
|
|
||||||
ArgProperty(title="是否允许从GPTAC学术云下载(或者上传)翻译结果(仅针对Arxiv论文)", options=["允许", "禁止"], default_value="禁止", description="共享文献，互助互利", type="dropdown").model_dump_json(),
|
|
||||||
}
|
|
||||||
return gui_definition
|
|
||||||
|
|
||||||
def execute(txt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_request):
|
|
||||||
"""
|
|
||||||
执行插件
|
|
||||||
"""
|
|
||||||
allow_cache = plugin_kwargs["allow_cache"]
|
|
||||||
allow_cloudio = plugin_kwargs["allow_cloudio"]
|
|
||||||
advanced_arg = plugin_kwargs["advanced_arg"]
|
|
||||||
|
|
||||||
if allow_cache == "从头执行": plugin_kwargs["advanced_arg"] = "--no-cache " + plugin_kwargs["advanced_arg"]
|
|
||||||
|
|
||||||
# 从云端下载翻译结果，以及上传翻译结果到云端；人人为我，我为人人。
|
|
||||||
if allow_cloudio == "允许": plugin_kwargs["advanced_arg"] = "--allow-cloudio " + plugin_kwargs["advanced_arg"]
|
|
||||||
|
|
||||||
yield from Latex翻译中文并重新编译PDF(txt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_request)
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
class PDF_Localize(GptAcademicPluginTemplate):
|
|
||||||
def __init__(self):
|
|
||||||
"""
|
|
||||||
请注意`execute`会执行在不同的线程中，因此您在定义和使用类变量时，应当慎之又慎！
|
|
||||||
"""
|
|
||||||
pass
|
|
||||||
|
|
||||||
def define_arg_selection_menu(self):
|
|
||||||
"""
|
|
||||||
定义插件的二级选项菜单
|
|
||||||
"""
|
|
||||||
gui_definition = {
|
|
||||||
"main_input":
|
|
||||||
ArgProperty(title="PDF文件路径", description="未指定路径，请上传文件后，再点击该插件", default_value="", type="string").model_dump_json(), # 主输入，自动从输入框同步
|
|
||||||
"advanced_arg":
|
|
||||||
ArgProperty(title="额外的翻译提示词",
|
|
||||||
description=r"如果有必要, 请在此处给出自定义翻译命令, 解决部分词汇翻译不准确的问题。 "
|
|
||||||
r"例如当单词'agent'翻译不准确时, 请尝试把以下指令复制到高级参数区: "
|
|
||||||
r'If the term "agent" is used in this section, it should be translated to "智能体". ',
|
|
||||||
default_value="", type="string").model_dump_json(), # 高级参数输入区，自动同步
|
|
||||||
"method":
|
|
||||||
ArgProperty(title="采用哪种方法执行转换", options=["MATHPIX", "DOC2X"], default_value="DOC2X", description="无", type="dropdown").model_dump_json(),
|
|
||||||
|
|
||||||
}
|
|
||||||
return gui_definition
|
|
||||||
|
|
||||||
def execute(txt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_request):
|
|
||||||
"""
|
|
||||||
执行插件
|
|
||||||
"""
|
|
||||||
yield from PDF翻译中文并重新编译PDF(txt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_request)
|
|
||||||
@@ -1,245 +0,0 @@
|
|||||||
from toolbox import update_ui, trimmed_format_exc, promote_file_to_downloadzone, get_log_folder
|
|
||||||
from toolbox import CatchException, report_exception, write_history_to_file, zip_folder
|
|
||||||
from loguru import logger
|
|
||||||
|
|
||||||
class PaperFileGroup():
|
|
||||||
def __init__(self):
|
|
||||||
self.file_paths = []
|
|
||||||
self.file_contents = []
|
|
||||||
self.sp_file_contents = []
|
|
||||||
self.sp_file_index = []
|
|
||||||
self.sp_file_tag = []
|
|
||||||
|
|
||||||
# count_token
|
|
||||||
from request_llms.bridge_all import model_info
|
|
||||||
enc = model_info["gpt-3.5-turbo"]['tokenizer']
|
|
||||||
def get_token_num(txt): return len(enc.encode(txt, disallowed_special=()))
|
|
||||||
self.get_token_num = get_token_num
|
|
||||||
|
|
||||||
def run_file_split(self, max_token_limit=1900):
|
|
||||||
"""
|
|
||||||
将长文本分离开来
|
|
||||||
"""
|
|
||||||
for index, file_content in enumerate(self.file_contents):
|
|
||||||
if self.get_token_num(file_content) < max_token_limit:
|
|
||||||
self.sp_file_contents.append(file_content)
|
|
||||||
self.sp_file_index.append(index)
|
|
||||||
self.sp_file_tag.append(self.file_paths[index])
|
|
||||||
else:
|
|
||||||
from crazy_functions.pdf_fns.breakdown_txt import breakdown_text_to_satisfy_token_limit
|
|
||||||
segments = breakdown_text_to_satisfy_token_limit(file_content, max_token_limit)
|
|
||||||
for j, segment in enumerate(segments):
|
|
||||||
self.sp_file_contents.append(segment)
|
|
||||||
self.sp_file_index.append(index)
|
|
||||||
self.sp_file_tag.append(self.file_paths[index] + f".part-{j}.tex")
|
|
||||||
|
|
||||||
logger.info('Segmentation: done')
|
|
||||||
def merge_result(self):
|
|
||||||
self.file_result = ["" for _ in range(len(self.file_paths))]
|
|
||||||
for r, k in zip(self.sp_file_result, self.sp_file_index):
|
|
||||||
self.file_result[k] += r
|
|
||||||
|
|
||||||
def write_result(self):
|
|
||||||
manifest = []
|
|
||||||
for path, res in zip(self.file_paths, self.file_result):
|
|
||||||
with open(path + '.polish.tex', 'w', encoding='utf8') as f:
|
|
||||||
manifest.append(path + '.polish.tex')
|
|
||||||
f.write(res)
|
|
||||||
return manifest
|
|
||||||
|
|
||||||
def zip_result(self):
|
|
||||||
import os, time
|
|
||||||
folder = os.path.dirname(self.file_paths[0])
|
|
||||||
t = time.strftime("%Y-%m-%d-%H-%M-%S", time.localtime())
|
|
||||||
zip_folder(folder, get_log_folder(), f'{t}-polished.zip')
|
|
||||||
|
|
||||||
|
|
||||||
def 多文件润色(file_manifest, project_folder, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, language='en', mode='polish'):
|
|
||||||
import time, os, re
|
|
||||||
from crazy_functions.crazy_utils import request_gpt_model_multi_threads_with_very_awesome_ui_and_high_efficiency
|
|
||||||
|
|
||||||
|
|
||||||
# <-------- 读取Latex文件，删除其中的所有注释 ---------->
|
|
||||||
pfg = PaperFileGroup()
|
|
||||||
|
|
||||||
for index, fp in enumerate(file_manifest):
|
|
||||||
with open(fp, 'r', encoding='utf-8', errors='replace') as f:
|
|
||||||
file_content = f.read()
|
|
||||||
# 定义注释的正则表达式
|
|
||||||
comment_pattern = r'(?<!\\)%.*'
|
|
||||||
# 使用正则表达式查找注释，并替换为空字符串
|
|
||||||
clean_tex_content = re.sub(comment_pattern, '', file_content)
|
|
||||||
# 记录删除注释后的文本
|
|
||||||
pfg.file_paths.append(fp)
|
|
||||||
pfg.file_contents.append(clean_tex_content)
|
|
||||||
|
|
||||||
# <-------- 拆分过长的latex文件 ---------->
|
|
||||||
pfg.run_file_split(max_token_limit=1024)
|
|
||||||
n_split = len(pfg.sp_file_contents)
|
|
||||||
|
|
||||||
|
|
||||||
# <-------- 多线程润色开始 ---------->
|
|
||||||
if language == 'en':
|
|
||||||
if mode == 'polish':
|
|
||||||
inputs_array = [r"Below is a section from an academic paper, polish this section to meet the academic standard, " +
|
|
||||||
r"improve the grammar, clarity and overall readability, do not modify any latex command such as \section, \cite and equations:" +
|
|
||||||
f"\n\n{frag}" for frag in pfg.sp_file_contents]
|
|
||||||
else:
|
|
||||||
inputs_array = [r"Below is a section from an academic paper, proofread this section." +
|
|
||||||
r"Do not modify any latex command such as \section, \cite, \begin, \item and equations. " +
|
|
||||||
r"Answer me only with the revised text:" +
|
|
||||||
f"\n\n{frag}" for frag in pfg.sp_file_contents]
|
|
||||||
inputs_show_user_array = [f"Polish {f}" for f in pfg.sp_file_tag]
|
|
||||||
sys_prompt_array = ["You are a professional academic paper writer." for _ in range(n_split)]
|
|
||||||
elif language == 'zh':
|
|
||||||
if mode == 'polish':
|
|
||||||
inputs_array = [r"以下是一篇学术论文中的一段内容，请将此部分润色以满足学术标准，提高语法、清晰度和整体可读性，不要修改任何LaTeX命令，例如\section，\cite和方程式：" +
|
|
||||||
f"\n\n{frag}" for frag in pfg.sp_file_contents]
|
|
||||||
else:
|
|
||||||
inputs_array = [r"以下是一篇学术论文中的一段内容，请对这部分内容进行语法矫正。不要修改任何LaTeX命令，例如\section，\cite和方程式：" +
|
|
||||||
f"\n\n{frag}" for frag in pfg.sp_file_contents]
|
|
||||||
inputs_show_user_array = [f"润色 {f}" for f in pfg.sp_file_tag]
|
|
||||||
sys_prompt_array=["你是一位专业的中文学术论文作家。" for _ in range(n_split)]
|
|
||||||
|
|
||||||
|
|
||||||
gpt_response_collection = yield from request_gpt_model_multi_threads_with_very_awesome_ui_and_high_efficiency(
|
|
||||||
inputs_array=inputs_array,
|
|
||||||
inputs_show_user_array=inputs_show_user_array,
|
|
||||||
llm_kwargs=llm_kwargs,
|
|
||||||
chatbot=chatbot,
|
|
||||||
history_array=[[""] for _ in range(n_split)],
|
|
||||||
sys_prompt_array=sys_prompt_array,
|
|
||||||
# max_workers=5, # 并行任务数量限制，最多同时执行5个，其他的排队等待
|
|
||||||
scroller_max_len = 80
|
|
||||||
)
|
|
||||||
|
|
||||||
# <-------- 文本碎片重组为完整的tex文件，整理结果为压缩包 ---------->
|
|
||||||
try:
|
|
||||||
pfg.sp_file_result = []
|
|
||||||
for i_say, gpt_say in zip(gpt_response_collection[0::2], gpt_response_collection[1::2]):
|
|
||||||
pfg.sp_file_result.append(gpt_say)
|
|
||||||
pfg.merge_result()
|
|
||||||
pfg.write_result()
|
|
||||||
pfg.zip_result()
|
|
||||||
except:
|
|
||||||
logger.error(trimmed_format_exc())
|
|
||||||
|
|
||||||
# <-------- 整理结果，退出 ---------->
|
|
||||||
create_report_file_name = time.strftime("%Y-%m-%d-%H-%M-%S", time.localtime()) + f"-chatgpt.polish.md"
|
|
||||||
res = write_history_to_file(gpt_response_collection, file_basename=create_report_file_name)
|
|
||||||
promote_file_to_downloadzone(res, chatbot=chatbot)
|
|
||||||
|
|
||||||
history = gpt_response_collection
|
|
||||||
chatbot.append((f"{fp}完成了吗？", res))
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
|
|
||||||
|
|
||||||
@CatchException
|
|
||||||
def Latex英文润色(txt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_request):
|
|
||||||
# 基本信息：功能、贡献者
|
|
||||||
chatbot.append([
|
|
||||||
"函数插件功能？",
|
|
||||||
"对整个Latex项目进行润色。函数插件贡献者: Binary-Husky。（注意，此插件不调用Latex，如果有Latex环境，请使用「Latex英文纠错+高亮修正位置(需Latex)插件」"])
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
|
|
||||||
# 尝试导入依赖，如果缺少依赖，则给出安装建议
|
|
||||||
try:
|
|
||||||
import tiktoken
|
|
||||||
except:
|
|
||||||
report_exception(chatbot, history,
|
|
||||||
a=f"解析项目: {txt}",
|
|
||||||
b=f"导入软件依赖失败。使用该模块需要额外依赖，安装方法```pip install --upgrade tiktoken```。")
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
history = [] # 清空历史，以免输入溢出
|
|
||||||
import glob, os
|
|
||||||
if os.path.exists(txt):
|
|
||||||
project_folder = txt
|
|
||||||
else:
|
|
||||||
if txt == "": txt = '空空如也的输入栏'
|
|
||||||
report_exception(chatbot, history, a = f"解析项目: {txt}", b = f"找不到本地项目或无权访问: {txt}")
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
file_manifest = [f for f in glob.glob(f'{project_folder}/**/*.tex', recursive=True)]
|
|
||||||
if len(file_manifest) == 0:
|
|
||||||
report_exception(chatbot, history, a = f"解析项目: {txt}", b = f"找不到任何.tex文件: {txt}")
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
yield from 多文件润色(file_manifest, project_folder, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, language='en')
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@CatchException
|
|
||||||
def Latex中文润色(txt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_request):
|
|
||||||
# 基本信息：功能、贡献者
|
|
||||||
chatbot.append([
|
|
||||||
"函数插件功能？",
|
|
||||||
"对整个Latex项目进行润色。函数插件贡献者: Binary-Husky"])
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
|
|
||||||
# 尝试导入依赖，如果缺少依赖，则给出安装建议
|
|
||||||
try:
|
|
||||||
import tiktoken
|
|
||||||
except:
|
|
||||||
report_exception(chatbot, history,
|
|
||||||
a=f"解析项目: {txt}",
|
|
||||||
b=f"导入软件依赖失败。使用该模块需要额外依赖，安装方法```pip install --upgrade tiktoken```。")
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
history = [] # 清空历史，以免输入溢出
|
|
||||||
import glob, os
|
|
||||||
if os.path.exists(txt):
|
|
||||||
project_folder = txt
|
|
||||||
else:
|
|
||||||
if txt == "": txt = '空空如也的输入栏'
|
|
||||||
report_exception(chatbot, history, a = f"解析项目: {txt}", b = f"找不到本地项目或无权访问: {txt}")
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
file_manifest = [f for f in glob.glob(f'{project_folder}/**/*.tex', recursive=True)]
|
|
||||||
if len(file_manifest) == 0:
|
|
||||||
report_exception(chatbot, history, a = f"解析项目: {txt}", b = f"找不到任何.tex文件: {txt}")
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
yield from 多文件润色(file_manifest, project_folder, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, language='zh')
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@CatchException
|
|
||||||
def Latex英文纠错(txt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_request):
|
|
||||||
# 基本信息：功能、贡献者
|
|
||||||
chatbot.append([
|
|
||||||
"函数插件功能？",
|
|
||||||
"对整个Latex项目进行纠错。函数插件贡献者: Binary-Husky"])
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
|
|
||||||
# 尝试导入依赖，如果缺少依赖，则给出安装建议
|
|
||||||
try:
|
|
||||||
import tiktoken
|
|
||||||
except:
|
|
||||||
report_exception(chatbot, history,
|
|
||||||
a=f"解析项目: {txt}",
|
|
||||||
b=f"导入软件依赖失败。使用该模块需要额外依赖，安装方法```pip install --upgrade tiktoken```。")
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
history = [] # 清空历史，以免输入溢出
|
|
||||||
import glob, os
|
|
||||||
if os.path.exists(txt):
|
|
||||||
project_folder = txt
|
|
||||||
else:
|
|
||||||
if txt == "": txt = '空空如也的输入栏'
|
|
||||||
report_exception(chatbot, history, a = f"解析项目: {txt}", b = f"找不到本地项目或无权访问: {txt}")
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
file_manifest = [f for f in glob.glob(f'{project_folder}/**/*.tex', recursive=True)]
|
|
||||||
if len(file_manifest) == 0:
|
|
||||||
report_exception(chatbot, history, a = f"解析项目: {txt}", b = f"找不到任何.tex文件: {txt}")
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
yield from 多文件润色(file_manifest, project_folder, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, language='en', mode='proofread')
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -1,176 +0,0 @@
|
|||||||
from toolbox import update_ui, promote_file_to_downloadzone
|
|
||||||
from toolbox import CatchException, report_exception, write_history_to_file
|
|
||||||
from loguru import logger
|
|
||||||
|
|
||||||
class PaperFileGroup():
|
|
||||||
def __init__(self):
|
|
||||||
self.file_paths = []
|
|
||||||
self.file_contents = []
|
|
||||||
self.sp_file_contents = []
|
|
||||||
self.sp_file_index = []
|
|
||||||
self.sp_file_tag = []
|
|
||||||
|
|
||||||
# count_token
|
|
||||||
from request_llms.bridge_all import model_info
|
|
||||||
enc = model_info["gpt-3.5-turbo"]['tokenizer']
|
|
||||||
def get_token_num(txt): return len(enc.encode(txt, disallowed_special=()))
|
|
||||||
self.get_token_num = get_token_num
|
|
||||||
|
|
||||||
def run_file_split(self, max_token_limit=1900):
|
|
||||||
"""
|
|
||||||
将长文本分离开来
|
|
||||||
"""
|
|
||||||
for index, file_content in enumerate(self.file_contents):
|
|
||||||
if self.get_token_num(file_content) < max_token_limit:
|
|
||||||
self.sp_file_contents.append(file_content)
|
|
||||||
self.sp_file_index.append(index)
|
|
||||||
self.sp_file_tag.append(self.file_paths[index])
|
|
||||||
else:
|
|
||||||
from crazy_functions.pdf_fns.breakdown_txt import breakdown_text_to_satisfy_token_limit
|
|
||||||
segments = breakdown_text_to_satisfy_token_limit(file_content, max_token_limit)
|
|
||||||
for j, segment in enumerate(segments):
|
|
||||||
self.sp_file_contents.append(segment)
|
|
||||||
self.sp_file_index.append(index)
|
|
||||||
self.sp_file_tag.append(self.file_paths[index] + f".part-{j}.tex")
|
|
||||||
|
|
||||||
logger.info('Segmentation: done')
|
|
||||||
|
|
||||||
def 多文件翻译(file_manifest, project_folder, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, language='en'):
|
|
||||||
import time, os, re
|
|
||||||
from crazy_functions.crazy_utils import request_gpt_model_multi_threads_with_very_awesome_ui_and_high_efficiency
|
|
||||||
|
|
||||||
# <-------- 读取Latex文件，删除其中的所有注释 ---------->
|
|
||||||
pfg = PaperFileGroup()
|
|
||||||
|
|
||||||
for index, fp in enumerate(file_manifest):
|
|
||||||
with open(fp, 'r', encoding='utf-8', errors='replace') as f:
|
|
||||||
file_content = f.read()
|
|
||||||
# 定义注释的正则表达式
|
|
||||||
comment_pattern = r'(?<!\\)%.*'
|
|
||||||
# 使用正则表达式查找注释，并替换为空字符串
|
|
||||||
clean_tex_content = re.sub(comment_pattern, '', file_content)
|
|
||||||
# 记录删除注释后的文本
|
|
||||||
pfg.file_paths.append(fp)
|
|
||||||
pfg.file_contents.append(clean_tex_content)
|
|
||||||
|
|
||||||
# <-------- 拆分过长的latex文件 ---------->
|
|
||||||
pfg.run_file_split(max_token_limit=1024)
|
|
||||||
n_split = len(pfg.sp_file_contents)
|
|
||||||
|
|
||||||
# <-------- 抽取摘要 ---------->
|
|
||||||
# if language == 'en':
|
|
||||||
# abs_extract_inputs = f"Please write an abstract for this paper"
|
|
||||||
|
|
||||||
# # 单线，获取文章meta信息
|
|
||||||
# paper_meta_info = yield from request_gpt_model_in_new_thread_with_ui_alive(
|
|
||||||
# inputs=abs_extract_inputs,
|
|
||||||
# inputs_show_user=f"正在抽取摘要信息。",
|
|
||||||
# llm_kwargs=llm_kwargs,
|
|
||||||
# chatbot=chatbot, history=[],
|
|
||||||
# sys_prompt="Your job is to collect information from materials。",
|
|
||||||
# )
|
|
||||||
|
|
||||||
# <-------- 多线程润色开始 ---------->
|
|
||||||
if language == 'en->zh':
|
|
||||||
inputs_array = ["Below is a section from an English academic paper, translate it into Chinese, do not modify any latex command such as \section, \cite and equations:" +
|
|
||||||
f"\n\n{frag}" for frag in pfg.sp_file_contents]
|
|
||||||
inputs_show_user_array = [f"翻译 {f}" for f in pfg.sp_file_tag]
|
|
||||||
sys_prompt_array = ["You are a professional academic paper translator." for _ in range(n_split)]
|
|
||||||
elif language == 'zh->en':
|
|
||||||
inputs_array = [f"Below is a section from a Chinese academic paper, translate it into English, do not modify any latex command such as \section, \cite and equations:" +
|
|
||||||
f"\n\n{frag}" for frag in pfg.sp_file_contents]
|
|
||||||
inputs_show_user_array = [f"翻译 {f}" for f in pfg.sp_file_tag]
|
|
||||||
sys_prompt_array = ["You are a professional academic paper translator." for _ in range(n_split)]
|
|
||||||
|
|
||||||
gpt_response_collection = yield from request_gpt_model_multi_threads_with_very_awesome_ui_and_high_efficiency(
|
|
||||||
inputs_array=inputs_array,
|
|
||||||
inputs_show_user_array=inputs_show_user_array,
|
|
||||||
llm_kwargs=llm_kwargs,
|
|
||||||
chatbot=chatbot,
|
|
||||||
history_array=[[""] for _ in range(n_split)],
|
|
||||||
sys_prompt_array=sys_prompt_array,
|
|
||||||
# max_workers=5, # OpenAI所允许的最大并行过载
|
|
||||||
scroller_max_len = 80
|
|
||||||
)
|
|
||||||
|
|
||||||
# <-------- 整理结果，退出 ---------->
|
|
||||||
create_report_file_name = time.strftime("%Y-%m-%d-%H-%M-%S", time.localtime()) + f"-chatgpt.polish.md"
|
|
||||||
res = write_history_to_file(gpt_response_collection, create_report_file_name)
|
|
||||||
promote_file_to_downloadzone(res, chatbot=chatbot)
|
|
||||||
history = gpt_response_collection
|
|
||||||
chatbot.append((f"{fp}完成了吗？", res))
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@CatchException
|
|
||||||
def Latex英译中(txt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_request):
|
|
||||||
# 基本信息：功能、贡献者
|
|
||||||
chatbot.append([
|
|
||||||
"函数插件功能？",
|
|
||||||
"对整个Latex项目进行翻译。函数插件贡献者: Binary-Husky"])
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
|
|
||||||
# 尝试导入依赖，如果缺少依赖，则给出安装建议
|
|
||||||
try:
|
|
||||||
import tiktoken
|
|
||||||
except:
|
|
||||||
report_exception(chatbot, history,
|
|
||||||
a=f"解析项目: {txt}",
|
|
||||||
b=f"导入软件依赖失败。使用该模块需要额外依赖，安装方法```pip install --upgrade tiktoken```。")
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
history = [] # 清空历史，以免输入溢出
|
|
||||||
import glob, os
|
|
||||||
if os.path.exists(txt):
|
|
||||||
project_folder = txt
|
|
||||||
else:
|
|
||||||
if txt == "": txt = '空空如也的输入栏'
|
|
||||||
report_exception(chatbot, history, a = f"解析项目: {txt}", b = f"找不到本地项目或无权访问: {txt}")
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
file_manifest = [f for f in glob.glob(f'{project_folder}/**/*.tex', recursive=True)]
|
|
||||||
if len(file_manifest) == 0:
|
|
||||||
report_exception(chatbot, history, a = f"解析项目: {txt}", b = f"找不到任何.tex文件: {txt}")
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
yield from 多文件翻译(file_manifest, project_folder, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, language='en->zh')
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@CatchException
|
|
||||||
def Latex中译英(txt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_request):
|
|
||||||
# 基本信息：功能、贡献者
|
|
||||||
chatbot.append([
|
|
||||||
"函数插件功能？",
|
|
||||||
"对整个Latex项目进行翻译。函数插件贡献者: Binary-Husky"])
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
|
|
||||||
# 尝试导入依赖，如果缺少依赖，则给出安装建议
|
|
||||||
try:
|
|
||||||
import tiktoken
|
|
||||||
except:
|
|
||||||
report_exception(chatbot, history,
|
|
||||||
a=f"解析项目: {txt}",
|
|
||||||
b=f"导入软件依赖失败。使用该模块需要额外依赖，安装方法```pip install --upgrade tiktoken```。")
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
history = [] # 清空历史，以免输入溢出
|
|
||||||
import glob, os
|
|
||||||
if os.path.exists(txt):
|
|
||||||
project_folder = txt
|
|
||||||
else:
|
|
||||||
if txt == "": txt = '空空如也的输入栏'
|
|
||||||
report_exception(chatbot, history, a = f"解析项目: {txt}", b = f"找不到本地项目或无权访问: {txt}")
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
file_manifest = [f for f in glob.glob(f'{project_folder}/**/*.tex', recursive=True)]
|
|
||||||
if len(file_manifest) == 0:
|
|
||||||
report_exception(chatbot, history, a = f"解析项目: {txt}", b = f"找不到任何.tex文件: {txt}")
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
yield from 多文件翻译(file_manifest, project_folder, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, language='zh->en')
|
|
||||||
@@ -1,264 +0,0 @@
|
|||||||
import glob, shutil, os, re
|
|
||||||
from loguru import logger
|
|
||||||
from toolbox import update_ui, trimmed_format_exc, gen_time_str
|
|
||||||
from toolbox import CatchException, report_exception, get_log_folder
|
|
||||||
from toolbox import write_history_to_file, promote_file_to_downloadzone
|
|
||||||
fast_debug = False
|
|
||||||
|
|
||||||
class PaperFileGroup():
|
|
||||||
def __init__(self):
|
|
||||||
self.file_paths = []
|
|
||||||
self.file_contents = []
|
|
||||||
self.sp_file_contents = []
|
|
||||||
self.sp_file_index = []
|
|
||||||
self.sp_file_tag = []
|
|
||||||
|
|
||||||
# count_token
|
|
||||||
from request_llms.bridge_all import model_info
|
|
||||||
enc = model_info["gpt-3.5-turbo"]['tokenizer']
|
|
||||||
def get_token_num(txt): return len(enc.encode(txt, disallowed_special=()))
|
|
||||||
self.get_token_num = get_token_num
|
|
||||||
|
|
||||||
def run_file_split(self, max_token_limit=2048):
|
|
||||||
"""
|
|
||||||
将长文本分离开来
|
|
||||||
"""
|
|
||||||
for index, file_content in enumerate(self.file_contents):
|
|
||||||
if self.get_token_num(file_content) < max_token_limit:
|
|
||||||
self.sp_file_contents.append(file_content)
|
|
||||||
self.sp_file_index.append(index)
|
|
||||||
self.sp_file_tag.append(self.file_paths[index])
|
|
||||||
else:
|
|
||||||
from crazy_functions.pdf_fns.breakdown_txt import breakdown_text_to_satisfy_token_limit
|
|
||||||
segments = breakdown_text_to_satisfy_token_limit(file_content, max_token_limit)
|
|
||||||
for j, segment in enumerate(segments):
|
|
||||||
self.sp_file_contents.append(segment)
|
|
||||||
self.sp_file_index.append(index)
|
|
||||||
self.sp_file_tag.append(self.file_paths[index] + f".part-{j}.md")
|
|
||||||
logger.info('Segmentation: done')
|
|
||||||
|
|
||||||
def merge_result(self):
|
|
||||||
self.file_result = ["" for _ in range(len(self.file_paths))]
|
|
||||||
for r, k in zip(self.sp_file_result, self.sp_file_index):
|
|
||||||
self.file_result[k] += r
|
|
||||||
|
|
||||||
def write_result(self, language):
|
|
||||||
manifest = []
|
|
||||||
for path, res in zip(self.file_paths, self.file_result):
|
|
||||||
dst_file = os.path.join(get_log_folder(), f'{gen_time_str()}.md')
|
|
||||||
with open(dst_file, 'w', encoding='utf8') as f:
|
|
||||||
manifest.append(dst_file)
|
|
||||||
f.write(res)
|
|
||||||
return manifest
|
|
||||||
|
|
||||||
def 多文件翻译(file_manifest, project_folder, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, language='en'):
|
|
||||||
from crazy_functions.crazy_utils import request_gpt_model_multi_threads_with_very_awesome_ui_and_high_efficiency
|
|
||||||
|
|
||||||
# <-------- 读取Markdown文件，删除其中的所有注释 ---------->
|
|
||||||
pfg = PaperFileGroup()
|
|
||||||
|
|
||||||
for index, fp in enumerate(file_manifest):
|
|
||||||
with open(fp, 'r', encoding='utf-8', errors='replace') as f:
|
|
||||||
file_content = f.read()
|
|
||||||
# 记录删除注释后的文本
|
|
||||||
pfg.file_paths.append(fp)
|
|
||||||
pfg.file_contents.append(file_content)
|
|
||||||
|
|
||||||
# <-------- 拆分过长的Markdown文件 ---------->
|
|
||||||
pfg.run_file_split(max_token_limit=1024)
|
|
||||||
n_split = len(pfg.sp_file_contents)
|
|
||||||
|
|
||||||
# <-------- 多线程翻译开始 ---------->
|
|
||||||
if language == 'en->zh':
|
|
||||||
inputs_array = ["This is a Markdown file, translate it into Chinese, do NOT modify any existing Markdown commands, do NOT use code wrapper (```), ONLY answer me with translated results:" +
|
|
||||||
f"\n\n{frag}" for frag in pfg.sp_file_contents]
|
|
||||||
inputs_show_user_array = [f"翻译 {f}" for f in pfg.sp_file_tag]
|
|
||||||
sys_prompt_array = ["You are a professional academic paper translator." + plugin_kwargs.get("additional_prompt", "") for _ in range(n_split)]
|
|
||||||
elif language == 'zh->en':
|
|
||||||
inputs_array = [f"This is a Markdown file, translate it into English, do NOT modify any existing Markdown commands, do NOT use code wrapper (```), ONLY answer me with translated results:" +
|
|
||||||
f"\n\n{frag}" for frag in pfg.sp_file_contents]
|
|
||||||
inputs_show_user_array = [f"翻译 {f}" for f in pfg.sp_file_tag]
|
|
||||||
sys_prompt_array = ["You are a professional academic paper translator." + plugin_kwargs.get("additional_prompt", "") for _ in range(n_split)]
|
|
||||||
else:
|
|
||||||
inputs_array = [f"This is a Markdown file, translate it into {language}, do NOT modify any existing Markdown commands, do NOT use code wrapper (```), ONLY answer me with translated results:" +
|
|
||||||
f"\n\n{frag}" for frag in pfg.sp_file_contents]
|
|
||||||
inputs_show_user_array = [f"翻译 {f}" for f in pfg.sp_file_tag]
|
|
||||||
sys_prompt_array = ["You are a professional academic paper translator." + plugin_kwargs.get("additional_prompt", "") for _ in range(n_split)]
|
|
||||||
|
|
||||||
gpt_response_collection = yield from request_gpt_model_multi_threads_with_very_awesome_ui_and_high_efficiency(
|
|
||||||
inputs_array=inputs_array,
|
|
||||||
inputs_show_user_array=inputs_show_user_array,
|
|
||||||
llm_kwargs=llm_kwargs,
|
|
||||||
chatbot=chatbot,
|
|
||||||
history_array=[[""] for _ in range(n_split)],
|
|
||||||
sys_prompt_array=sys_prompt_array,
|
|
||||||
# max_workers=5, # OpenAI所允许的最大并行过载
|
|
||||||
scroller_max_len = 80
|
|
||||||
)
|
|
||||||
try:
|
|
||||||
pfg.sp_file_result = []
|
|
||||||
for i_say, gpt_say in zip(gpt_response_collection[0::2], gpt_response_collection[1::2]):
|
|
||||||
pfg.sp_file_result.append(gpt_say)
|
|
||||||
pfg.merge_result()
|
|
||||||
output_file_arr = pfg.write_result(language)
|
|
||||||
for output_file in output_file_arr:
|
|
||||||
promote_file_to_downloadzone(output_file, chatbot=chatbot)
|
|
||||||
if 'markdown_expected_output_path' in plugin_kwargs:
|
|
||||||
expected_f_name = plugin_kwargs['markdown_expected_output_path']
|
|
||||||
shutil.copyfile(output_file, expected_f_name)
|
|
||||||
except:
|
|
||||||
logger.error(trimmed_format_exc())
|
|
||||||
|
|
||||||
# <-------- 整理结果，退出 ---------->
|
|
||||||
create_report_file_name = gen_time_str() + f"-chatgpt.md"
|
|
||||||
res = write_history_to_file(gpt_response_collection, file_basename=create_report_file_name)
|
|
||||||
promote_file_to_downloadzone(res, chatbot=chatbot)
|
|
||||||
history = gpt_response_collection
|
|
||||||
chatbot.append((f"{fp}完成了吗？", res))
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
|
|
||||||
|
|
||||||
def get_files_from_everything(txt, preference=''):
|
|
||||||
if txt == "": return False, None, None
|
|
||||||
success = True
|
|
||||||
if txt.startswith('http'):
|
|
||||||
import requests
|
|
||||||
from toolbox import get_conf
|
|
||||||
proxies = get_conf('proxies')
|
|
||||||
# 网络的远程文件
|
|
||||||
if preference == 'Github':
|
|
||||||
logger.info('正在从github下载资源 ...')
|
|
||||||
if not txt.endswith('.md'):
|
|
||||||
# Make a request to the GitHub API to retrieve the repository information
|
|
||||||
url = txt.replace("https://github.com/", "https://api.github.com/repos/") + '/readme'
|
|
||||||
response = requests.get(url, proxies=proxies)
|
|
||||||
txt = response.json()['download_url']
|
|
||||||
else:
|
|
||||||
txt = txt.replace("https://github.com/", "https://raw.githubusercontent.com/")
|
|
||||||
txt = txt.replace("/blob/", "/")
|
|
||||||
|
|
||||||
r = requests.get(txt, proxies=proxies)
|
|
||||||
download_local = f'{get_log_folder(plugin_name="批量Markdown翻译")}/raw-readme-{gen_time_str()}.md'
|
|
||||||
project_folder = f'{get_log_folder(plugin_name="批量Markdown翻译")}'
|
|
||||||
with open(download_local, 'wb+') as f: f.write(r.content)
|
|
||||||
file_manifest = [download_local]
|
|
||||||
elif txt.endswith('.md'):
|
|
||||||
# 直接给定文件
|
|
||||||
file_manifest = [txt]
|
|
||||||
project_folder = os.path.dirname(txt)
|
|
||||||
elif os.path.exists(txt):
|
|
||||||
# 本地路径，递归搜索
|
|
||||||
project_folder = txt
|
|
||||||
file_manifest = [f for f in glob.glob(f'{project_folder}/**/*.md', recursive=True)]
|
|
||||||
else:
|
|
||||||
project_folder = None
|
|
||||||
file_manifest = []
|
|
||||||
success = False
|
|
||||||
|
|
||||||
return success, file_manifest, project_folder
|
|
||||||
|
|
||||||
|
|
||||||
@CatchException
|
|
||||||
def Markdown英译中(txt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_request):
|
|
||||||
# 基本信息：功能、贡献者
|
|
||||||
chatbot.append([
|
|
||||||
"函数插件功能？",
|
|
||||||
"对整个Markdown项目进行翻译。函数插件贡献者: Binary-Husky"])
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
|
|
||||||
# 尝试导入依赖，如果缺少依赖，则给出安装建议
|
|
||||||
try:
|
|
||||||
import tiktoken
|
|
||||||
except:
|
|
||||||
report_exception(chatbot, history,
|
|
||||||
a=f"解析项目: {txt}",
|
|
||||||
b=f"导入软件依赖失败。使用该模块需要额外依赖，安装方法```pip install --upgrade tiktoken```。")
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
history = [] # 清空历史，以免输入溢出
|
|
||||||
|
|
||||||
success, file_manifest, project_folder = get_files_from_everything(txt, preference="Github")
|
|
||||||
|
|
||||||
if not success:
|
|
||||||
# 什么都没有
|
|
||||||
if txt == "": txt = '空空如也的输入栏'
|
|
||||||
report_exception(chatbot, history, a = f"解析项目: {txt}", b = f"找不到本地项目或无权访问: {txt}")
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
|
|
||||||
if len(file_manifest) == 0:
|
|
||||||
report_exception(chatbot, history, a = f"解析项目: {txt}", b = f"找不到任何.md文件: {txt}")
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
|
|
||||||
yield from 多文件翻译(file_manifest, project_folder, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, language='en->zh')
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@CatchException
|
|
||||||
def Markdown中译英(txt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_request):
|
|
||||||
# 基本信息：功能、贡献者
|
|
||||||
chatbot.append([
|
|
||||||
"函数插件功能？",
|
|
||||||
"对整个Markdown项目进行翻译。函数插件贡献者: Binary-Husky"])
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
|
|
||||||
# 尝试导入依赖，如果缺少依赖，则给出安装建议
|
|
||||||
try:
|
|
||||||
import tiktoken
|
|
||||||
except:
|
|
||||||
report_exception(chatbot, history,
|
|
||||||
a=f"解析项目: {txt}",
|
|
||||||
b=f"导入软件依赖失败。使用该模块需要额外依赖，安装方法```pip install --upgrade tiktoken```。")
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
history = [] # 清空历史，以免输入溢出
|
|
||||||
success, file_manifest, project_folder = get_files_from_everything(txt)
|
|
||||||
if not success:
|
|
||||||
# 什么都没有
|
|
||||||
if txt == "": txt = '空空如也的输入栏'
|
|
||||||
report_exception(chatbot, history, a = f"解析项目: {txt}", b = f"找不到本地项目或无权访问: {txt}")
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
if len(file_manifest) == 0:
|
|
||||||
report_exception(chatbot, history, a = f"解析项目: {txt}", b = f"找不到任何.md文件: {txt}")
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
yield from 多文件翻译(file_manifest, project_folder, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, language='zh->en')
|
|
||||||
|
|
||||||
|
|
||||||
@CatchException
|
|
||||||
def Markdown翻译指定语言(txt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_request):
|
|
||||||
# 基本信息：功能、贡献者
|
|
||||||
chatbot.append([
|
|
||||||
"函数插件功能？",
|
|
||||||
"对整个Markdown项目进行翻译。函数插件贡献者: Binary-Husky"])
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
|
|
||||||
# 尝试导入依赖，如果缺少依赖，则给出安装建议
|
|
||||||
try:
|
|
||||||
import tiktoken
|
|
||||||
except:
|
|
||||||
report_exception(chatbot, history,
|
|
||||||
a=f"解析项目: {txt}",
|
|
||||||
b=f"导入软件依赖失败。使用该模块需要额外依赖，安装方法```pip install --upgrade tiktoken```。")
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
history = [] # 清空历史，以免输入溢出
|
|
||||||
success, file_manifest, project_folder = get_files_from_everything(txt)
|
|
||||||
if not success:
|
|
||||||
# 什么都没有
|
|
||||||
if txt == "": txt = '空空如也的输入栏'
|
|
||||||
report_exception(chatbot, history, a = f"解析项目: {txt}", b = f"找不到本地项目或无权访问: {txt}")
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
if len(file_manifest) == 0:
|
|
||||||
report_exception(chatbot, history, a = f"解析项目: {txt}", b = f"找不到任何.md文件: {txt}")
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
|
|
||||||
if ("advanced_arg" in plugin_kwargs) and (plugin_kwargs["advanced_arg"] == ""): plugin_kwargs.pop("advanced_arg")
|
|
||||||
language = plugin_kwargs.get("advanced_arg", 'Chinese')
|
|
||||||
yield from 多文件翻译(file_manifest, project_folder, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, language=language)
|
|
||||||
@@ -1,83 +0,0 @@
|
|||||||
from toolbox import CatchException, check_packages, get_conf
|
|
||||||
from toolbox import update_ui, update_ui_lastest_msg, disable_auto_promotion
|
|
||||||
from toolbox import trimmed_format_exc_markdown
|
|
||||||
from crazy_functions.crazy_utils import get_files_from_everything
|
|
||||||
from crazy_functions.pdf_fns.parse_pdf import get_avail_grobid_url
|
|
||||||
from crazy_functions.pdf_fns.parse_pdf_via_doc2x import 解析PDF_基于DOC2X
|
|
||||||
from crazy_functions.pdf_fns.parse_pdf_legacy import 解析PDF_简单拆解
|
|
||||||
from crazy_functions.pdf_fns.parse_pdf_grobid import 解析PDF_基于GROBID
|
|
||||||
from shared_utils.colorful import *
|
|
||||||
|
|
||||||
@CatchException
|
|
||||||
def 批量翻译PDF文档(txt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_request):
|
|
||||||
|
|
||||||
disable_auto_promotion(chatbot)
|
|
||||||
# 基本信息：功能、贡献者
|
|
||||||
chatbot.append([None, "插件功能：批量翻译PDF文档。函数插件贡献者: Binary-Husky"])
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
|
|
||||||
# 尝试导入依赖，如果缺少依赖，则给出安装建议
|
|
||||||
try:
|
|
||||||
check_packages(["fitz", "tiktoken", "scipdf"])
|
|
||||||
except:
|
|
||||||
chatbot.append([None, f"导入软件依赖失败。使用该模块需要额外依赖，安装方法```pip install --upgrade pymupdf tiktoken scipdf_parser```。"])
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
|
|
||||||
# 清空历史，以免输入溢出
|
|
||||||
history = []
|
|
||||||
success, file_manifest, project_folder = get_files_from_everything(txt, type='.pdf')
|
|
||||||
|
|
||||||
# 检测输入参数，如没有给定输入参数，直接退出
|
|
||||||
if (not success) and txt == "": txt = '空空如也的输入栏。提示：请先上传文件（把PDF文件拖入对话）。'
|
|
||||||
|
|
||||||
# 如果没找到任何文件
|
|
||||||
if len(file_manifest) == 0:
|
|
||||||
chatbot.append([None, f"找不到任何.pdf拓展名的文件: {txt}"])
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
|
|
||||||
# 开始正式执行任务
|
|
||||||
method = plugin_kwargs.get("pdf_parse_method", None)
|
|
||||||
if method == "DOC2X":
|
|
||||||
# ------- 第一种方法，效果最好，但是需要DOC2X服务 -------
|
|
||||||
DOC2X_API_KEY = get_conf("DOC2X_API_KEY")
|
|
||||||
if len(DOC2X_API_KEY) != 0:
|
|
||||||
try:
|
|
||||||
yield from 解析PDF_基于DOC2X(file_manifest, project_folder, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, DOC2X_API_KEY, user_request)
|
|
||||||
return
|
|
||||||
except:
|
|
||||||
chatbot.append([None, f"DOC2X服务不可用，请检查报错详细。{trimmed_format_exc_markdown()}"])
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history)
|
|
||||||
|
|
||||||
if method == "GROBID":
|
|
||||||
# ------- 第二种方法，效果次优 -------
|
|
||||||
grobid_url = get_avail_grobid_url()
|
|
||||||
if grobid_url is not None:
|
|
||||||
yield from 解析PDF_基于GROBID(file_manifest, project_folder, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, grobid_url)
|
|
||||||
return
|
|
||||||
|
|
||||||
if method == "ClASSIC":
|
|
||||||
# ------- 第三种方法，早期代码，效果不理想 -------
|
|
||||||
yield from update_ui_lastest_msg("GROBID服务不可用，请检查config中的GROBID_URL。作为替代，现在将执行效果稍差的旧版代码。", chatbot, history, delay=3)
|
|
||||||
yield from 解析PDF_简单拆解(file_manifest, project_folder, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt)
|
|
||||||
return
|
|
||||||
|
|
||||||
if method is None:
|
|
||||||
# ------- 以上三种方法都试一遍 -------
|
|
||||||
DOC2X_API_KEY = get_conf("DOC2X_API_KEY")
|
|
||||||
if len(DOC2X_API_KEY) != 0:
|
|
||||||
try:
|
|
||||||
yield from 解析PDF_基于DOC2X(file_manifest, project_folder, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, DOC2X_API_KEY, user_request)
|
|
||||||
return
|
|
||||||
except:
|
|
||||||
chatbot.append([None, f"DOC2X服务不可用，正在尝试GROBID。{trimmed_format_exc_markdown()}"])
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history)
|
|
||||||
grobid_url = get_avail_grobid_url()
|
|
||||||
if grobid_url is not None:
|
|
||||||
yield from 解析PDF_基于GROBID(file_manifest, project_folder, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, grobid_url)
|
|
||||||
return
|
|
||||||
yield from update_ui_lastest_msg("GROBID服务不可用，请检查config中的GROBID_URL。作为替代，现在将执行效果稍差的旧版代码。", chatbot, history, delay=3)
|
|
||||||
yield from 解析PDF_简单拆解(file_manifest, project_folder, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt)
|
|
||||||
return
|
|
||||||
|
|
||||||
@@ -1,33 +0,0 @@
|
|||||||
from crazy_functions.plugin_template.plugin_class_template import GptAcademicPluginTemplate, ArgProperty
|
|
||||||
from .PDF_Translate import 批量翻译PDF文档
|
|
||||||
|
|
||||||
|
|
||||||
class PDF_Tran(GptAcademicPluginTemplate):
|
|
||||||
def __init__(self):
|
|
||||||
"""
|
|
||||||
请注意`execute`会执行在不同的线程中，因此您在定义和使用类变量时，应当慎之又慎！
|
|
||||||
"""
|
|
||||||
pass
|
|
||||||
|
|
||||||
def define_arg_selection_menu(self):
|
|
||||||
"""
|
|
||||||
定义插件的二级选项菜单
|
|
||||||
"""
|
|
||||||
gui_definition = {
|
|
||||||
"main_input":
|
|
||||||
ArgProperty(title="PDF文件路径", description="未指定路径，请上传文件后，再点击该插件", default_value="", type="string").model_dump_json(), # 主输入，自动从输入框同步
|
|
||||||
"additional_prompt":
|
|
||||||
ArgProperty(title="额外提示词", description="例如：对专有名词、翻译语气等方面的要求", default_value="", type="string").model_dump_json(), # 高级参数输入区，自动同步
|
|
||||||
"pdf_parse_method":
|
|
||||||
ArgProperty(title="PDF解析方法", options=["DOC2X", "GROBID", "ClASSIC"], description="无", default_value="GROBID", type="dropdown").model_dump_json(),
|
|
||||||
}
|
|
||||||
return gui_definition
|
|
||||||
|
|
||||||
def execute(txt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_request):
|
|
||||||
"""
|
|
||||||
执行插件
|
|
||||||
"""
|
|
||||||
main_input = plugin_kwargs["main_input"]
|
|
||||||
additional_prompt = plugin_kwargs["additional_prompt"]
|
|
||||||
pdf_parse_method = plugin_kwargs["pdf_parse_method"]
|
|
||||||
yield from 批量翻译PDF文档(txt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_request)
|
|
||||||
@@ -1,153 +0,0 @@
|
|||||||
import os,glob
|
|
||||||
from typing import List
|
|
||||||
|
|
||||||
from shared_utils.fastapi_server import validate_path_safety
|
|
||||||
|
|
||||||
from toolbox import report_exception
|
|
||||||
from toolbox import CatchException, update_ui, get_conf, get_log_folder, update_ui_lastest_msg
|
|
||||||
from shared_utils.fastapi_server import validate_path_safety
|
|
||||||
from crazy_functions.crazy_utils import input_clipping
|
|
||||||
from crazy_functions.crazy_utils import request_gpt_model_in_new_thread_with_ui_alive
|
|
||||||
|
|
||||||
RAG_WORKER_REGISTER = {}
|
|
||||||
MAX_HISTORY_ROUND = 5
|
|
||||||
MAX_CONTEXT_TOKEN_LIMIT = 4096
|
|
||||||
REMEMBER_PREVIEW = 1000
|
|
||||||
|
|
||||||
@CatchException
|
|
||||||
def handle_document_upload(files: List[str], llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_request, rag_worker):
|
|
||||||
"""
|
|
||||||
Handles document uploads by extracting text and adding it to the vector store.
|
|
||||||
"""
|
|
||||||
from llama_index.core import Document
|
|
||||||
from crazy_functions.rag_fns.rag_file_support import extract_text, supports_format
|
|
||||||
user_name = chatbot.get_user()
|
|
||||||
checkpoint_dir = get_log_folder(user_name, plugin_name='experimental_rag')
|
|
||||||
|
|
||||||
for file_path in files:
|
|
||||||
try:
|
|
||||||
validate_path_safety(file_path, user_name)
|
|
||||||
text = extract_text(file_path)
|
|
||||||
if text is None:
|
|
||||||
chatbot.append(
|
|
||||||
[f"上传文件: {os.path.basename(file_path)}", f"文件解析失败，无法提取文本内容，请更换文件。失败原因可能为：1.文档格式过于复杂；2. 不支持的文件格式，支持的文件格式后缀有:" + ", ".join(supports_format)])
|
|
||||||
else:
|
|
||||||
chatbot.append(
|
|
||||||
[f"上传文件: {os.path.basename(file_path)}", f"上传文件前50个字符为:{text[:50]}。"])
|
|
||||||
document = Document(text=text, metadata={"source": file_path})
|
|
||||||
rag_worker.add_documents_to_vector_store([document])
|
|
||||||
chatbot.append([f"上传文件: {os.path.basename(file_path)}", "文件已成功添加到知识库。"])
|
|
||||||
except Exception as e:
|
|
||||||
report_exception(chatbot, history, a=f"处理文件: {file_path}", b=str(e))
|
|
||||||
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
# Main Q&A function with document upload support
|
|
||||||
@CatchException
|
|
||||||
def Rag问答(txt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_request):
|
|
||||||
|
|
||||||
# import vector store lib
|
|
||||||
VECTOR_STORE_TYPE = "Milvus"
|
|
||||||
if VECTOR_STORE_TYPE == "Milvus":
|
|
||||||
try:
|
|
||||||
from crazy_functions.rag_fns.milvus_worker import MilvusRagWorker as LlamaIndexRagWorker
|
|
||||||
except:
|
|
||||||
VECTOR_STORE_TYPE = "Simple"
|
|
||||||
if VECTOR_STORE_TYPE == "Simple":
|
|
||||||
from crazy_functions.rag_fns.llama_index_worker import LlamaIndexRagWorker
|
|
||||||
|
|
||||||
# 1. we retrieve rag worker from global context
|
|
||||||
user_name = chatbot.get_user()
|
|
||||||
checkpoint_dir = get_log_folder(user_name, plugin_name='experimental_rag')
|
|
||||||
|
|
||||||
if user_name in RAG_WORKER_REGISTER:
|
|
||||||
rag_worker = RAG_WORKER_REGISTER[user_name]
|
|
||||||
else:
|
|
||||||
rag_worker = RAG_WORKER_REGISTER[user_name] = LlamaIndexRagWorker(
|
|
||||||
user_name,
|
|
||||||
llm_kwargs,
|
|
||||||
checkpoint_dir=checkpoint_dir,
|
|
||||||
auto_load_checkpoint=True
|
|
||||||
)
|
|
||||||
|
|
||||||
current_context = f"{VECTOR_STORE_TYPE} @ {checkpoint_dir}"
|
|
||||||
tip = "提示：输入“清空向量数据库”可以清空RAG向量数据库"
|
|
||||||
|
|
||||||
# 2. Handle special commands
|
|
||||||
if os.path.exists(txt) and os.path.isdir(txt):
|
|
||||||
project_folder = txt
|
|
||||||
validate_path_safety(project_folder, chatbot.get_user())
|
|
||||||
# Extract file paths from the user input
|
|
||||||
# Assuming the user inputs file paths separated by commas after the command
|
|
||||||
file_paths = [f for f in glob.glob(f'{project_folder}/**/*', recursive=True)]
|
|
||||||
chatbot.append([txt, f'正在处理上传的文档 ({current_context}) ...'])
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
|
|
||||||
yield from handle_document_upload(file_paths, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_request, rag_worker)
|
|
||||||
return
|
|
||||||
|
|
||||||
elif txt == "清空向量数据库":
|
|
||||||
chatbot.append([txt, f'正在清空 ({current_context}) ...'])
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
rag_worker.purge_vector_store()
|
|
||||||
yield from update_ui_lastest_msg('已清空', chatbot, history, delay=0) # 刷新界面
|
|
||||||
return
|
|
||||||
|
|
||||||
# 3. Normal Q&A processing
|
|
||||||
chatbot.append([txt, f'正在召回知识 ({current_context}) ...'])
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
|
|
||||||
# 4. Clip history to reduce token consumption
|
|
||||||
txt_origin = txt
|
|
||||||
|
|
||||||
if len(history) > MAX_HISTORY_ROUND * 2:
|
|
||||||
history = history[-(MAX_HISTORY_ROUND * 2):]
|
|
||||||
txt_clip, history, flags = input_clipping(txt, history, max_token_limit=MAX_CONTEXT_TOKEN_LIMIT, return_clip_flags=True)
|
|
||||||
input_is_clipped_flag = (flags["original_input_len"] != flags["clipped_input_len"])
|
|
||||||
|
|
||||||
# 5. If input is clipped, add input to vector store before retrieve
|
|
||||||
if input_is_clipped_flag:
|
|
||||||
yield from update_ui_lastest_msg('检测到长输入, 正在向量化 ...', chatbot, history, delay=0) # 刷新界面
|
|
||||||
# Save input to vector store
|
|
||||||
rag_worker.add_text_to_vector_store(txt_origin)
|
|
||||||
yield from update_ui_lastest_msg('向量化完成 ...', chatbot, history, delay=0) # 刷新界面
|
|
||||||
|
|
||||||
if len(txt_origin) > REMEMBER_PREVIEW:
|
|
||||||
HALF = REMEMBER_PREVIEW // 2
|
|
||||||
i_say_to_remember = txt[:HALF] + f" ...\n...(省略{len(txt_origin)-REMEMBER_PREVIEW}字)...\n... " + txt[-HALF:]
|
|
||||||
if (flags["original_input_len"] - flags["clipped_input_len"]) > HALF:
|
|
||||||
txt_clip = txt_clip + f" ...\n...(省略{len(txt_origin)-len(txt_clip)-HALF}字)...\n... " + txt[-HALF:]
|
|
||||||
else:
|
|
||||||
i_say_to_remember = i_say = txt_clip
|
|
||||||
else:
|
|
||||||
i_say_to_remember = i_say = txt_clip
|
|
||||||
|
|
||||||
# 6. Search vector store and build prompts
|
|
||||||
nodes = rag_worker.retrieve_from_store_with_query(i_say)
|
|
||||||
prompt = rag_worker.build_prompt(query=i_say, nodes=nodes)
|
|
||||||
# 7. Query language model
|
|
||||||
if len(chatbot) != 0:
|
|
||||||
chatbot.pop(-1) # Pop temp chat, because we are going to add them again inside `request_gpt_model_in_new_thread_with_ui_alive`
|
|
||||||
|
|
||||||
model_say = yield from request_gpt_model_in_new_thread_with_ui_alive(
|
|
||||||
inputs=prompt,
|
|
||||||
inputs_show_user=i_say,
|
|
||||||
llm_kwargs=llm_kwargs,
|
|
||||||
chatbot=chatbot,
|
|
||||||
history=history,
|
|
||||||
sys_prompt=system_prompt,
|
|
||||||
retry_times_at_unknown_error=0
|
|
||||||
)
|
|
||||||
|
|
||||||
# 8. Remember Q&A
|
|
||||||
yield from update_ui_lastest_msg(
|
|
||||||
model_say + '</br></br>' + f'对话记忆中, 请稍等 ({current_context}) ...',
|
|
||||||
chatbot, history, delay=0.5
|
|
||||||
)
|
|
||||||
rag_worker.remember_qa(i_say_to_remember, model_say)
|
|
||||||
history.extend([i_say, model_say])
|
|
||||||
|
|
||||||
# 9. Final UI Update
|
|
||||||
yield from update_ui_lastest_msg(model_say, chatbot, history, delay=0, msg=tip)
|
|
||||||
@@ -1,167 +0,0 @@
|
|||||||
import pickle, os, random
|
|
||||||
from toolbox import CatchException, update_ui, get_conf, get_log_folder, update_ui_lastest_msg
|
|
||||||
from crazy_functions.crazy_utils import input_clipping
|
|
||||||
from crazy_functions.crazy_utils import request_gpt_model_in_new_thread_with_ui_alive
|
|
||||||
from request_llms.bridge_all import predict_no_ui_long_connection
|
|
||||||
from crazy_functions.json_fns.select_tool import structure_output, select_tool
|
|
||||||
from pydantic import BaseModel, Field
|
|
||||||
from loguru import logger
|
|
||||||
from typing import List
|
|
||||||
|
|
||||||
|
|
||||||
SOCIAL_NETWOK_WORKER_REGISTER = {}
|
|
||||||
|
|
||||||
class SocialNetwork():
|
|
||||||
def __init__(self):
|
|
||||||
self.people = []
|
|
||||||
|
|
||||||
class SaveAndLoad():
|
|
||||||
def __init__(self, user_name, llm_kwargs, auto_load_checkpoint=True, checkpoint_dir=None) -> None:
|
|
||||||
self.user_name = user_name
|
|
||||||
self.checkpoint_dir = checkpoint_dir
|
|
||||||
if auto_load_checkpoint:
|
|
||||||
self.social_network = self.load_from_checkpoint(checkpoint_dir)
|
|
||||||
else:
|
|
||||||
self.social_network = SocialNetwork()
|
|
||||||
|
|
||||||
def does_checkpoint_exist(self, checkpoint_dir=None):
|
|
||||||
import os, glob
|
|
||||||
if checkpoint_dir is None: checkpoint_dir = self.checkpoint_dir
|
|
||||||
if not os.path.exists(checkpoint_dir): return False
|
|
||||||
if len(glob.glob(os.path.join(checkpoint_dir, "social_network.pkl"))) == 0: return False
|
|
||||||
return True
|
|
||||||
|
|
||||||
def save_to_checkpoint(self, checkpoint_dir=None):
|
|
||||||
if checkpoint_dir is None: checkpoint_dir = self.checkpoint_dir
|
|
||||||
with open(os.path.join(checkpoint_dir, 'social_network.pkl'), "wb+") as f:
|
|
||||||
pickle.dump(self.social_network, f)
|
|
||||||
return
|
|
||||||
|
|
||||||
def load_from_checkpoint(self, checkpoint_dir=None):
|
|
||||||
if checkpoint_dir is None: checkpoint_dir = self.checkpoint_dir
|
|
||||||
if self.does_checkpoint_exist(checkpoint_dir=checkpoint_dir):
|
|
||||||
with open(os.path.join(checkpoint_dir, 'social_network.pkl'), "rb") as f:
|
|
||||||
social_network = pickle.load(f)
|
|
||||||
return social_network
|
|
||||||
else:
|
|
||||||
return SocialNetwork()
|
|
||||||
|
|
||||||
|
|
||||||
class Friend(BaseModel):
|
|
||||||
friend_name: str = Field(description="name of a friend")
|
|
||||||
friend_description: str = Field(description="description of a friend (everything about this friend)")
|
|
||||||
friend_relationship: str = Field(description="The relationship with a friend (e.g. friend, family, colleague)")
|
|
||||||
|
|
||||||
class FriendList(BaseModel):
|
|
||||||
friends_list: List[Friend] = Field(description="The list of friends")
|
|
||||||
|
|
||||||
|
|
||||||
class SocialNetworkWorker(SaveAndLoad):
|
|
||||||
def ai_socail_advice(self, prompt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, run_gpt_fn, intention_type):
|
|
||||||
pass
|
|
||||||
|
|
||||||
def ai_remove_friend(self, prompt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, run_gpt_fn, intention_type):
|
|
||||||
pass
|
|
||||||
|
|
||||||
def ai_list_friends(self, prompt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, run_gpt_fn, intention_type):
|
|
||||||
pass
|
|
||||||
|
|
||||||
def ai_add_multi_friends(self, prompt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, run_gpt_fn, intention_type):
|
|
||||||
friend, err_msg = structure_output(
|
|
||||||
txt=prompt,
|
|
||||||
prompt="根据提示, 解析多个联系人的身份信息\n\n",
|
|
||||||
err_msg=f"不能理解该联系人",
|
|
||||||
run_gpt_fn=run_gpt_fn,
|
|
||||||
pydantic_cls=FriendList
|
|
||||||
)
|
|
||||||
if friend.friends_list:
|
|
||||||
for f in friend.friends_list:
|
|
||||||
self.add_friend(f)
|
|
||||||
msg = f"成功添加{len(friend.friends_list)}个联系人: {str(friend.friends_list)}"
|
|
||||||
yield from update_ui_lastest_msg(lastmsg=msg, chatbot=chatbot, history=history, delay=0)
|
|
||||||
|
|
||||||
|
|
||||||
def run(self, txt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_request):
|
|
||||||
prompt = txt
|
|
||||||
run_gpt_fn = lambda inputs, sys_prompt: predict_no_ui_long_connection(inputs=inputs, llm_kwargs=llm_kwargs, history=[], sys_prompt=sys_prompt, observe_window=[])
|
|
||||||
self.tools_to_select = {
|
|
||||||
"SocialAdvice":{
|
|
||||||
"explain_to_llm": "如果用户希望获取社交指导，调用SocialAdvice生成一些社交建议",
|
|
||||||
"callback": self.ai_socail_advice,
|
|
||||||
},
|
|
||||||
"AddFriends":{
|
|
||||||
"explain_to_llm": "如果用户给出了联系人，调用AddMultiFriends把联系人添加到数据库",
|
|
||||||
"callback": self.ai_add_multi_friends,
|
|
||||||
},
|
|
||||||
"RemoveFriend":{
|
|
||||||
"explain_to_llm": "如果用户希望移除某个联系人，调用RemoveFriend",
|
|
||||||
"callback": self.ai_remove_friend,
|
|
||||||
},
|
|
||||||
"ListFriends":{
|
|
||||||
"explain_to_llm": "如果用户列举联系人，调用ListFriends",
|
|
||||||
"callback": self.ai_list_friends,
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
try:
|
|
||||||
Explaination = '\n'.join([f'{k}: {v["explain_to_llm"]}' for k, v in self.tools_to_select.items()])
|
|
||||||
class UserSociaIntention(BaseModel):
|
|
||||||
intention_type: str = Field(
|
|
||||||
description=
|
|
||||||
f"The type of user intention. You must choose from {self.tools_to_select.keys()}.\n\n"
|
|
||||||
f"Explaination:\n{Explaination}",
|
|
||||||
default="SocialAdvice"
|
|
||||||
)
|
|
||||||
pydantic_cls_instance, err_msg = select_tool(
|
|
||||||
prompt=txt,
|
|
||||||
run_gpt_fn=run_gpt_fn,
|
|
||||||
pydantic_cls=UserSociaIntention
|
|
||||||
)
|
|
||||||
except Exception as e:
|
|
||||||
yield from update_ui_lastest_msg(
|
|
||||||
lastmsg=f"无法理解用户意图 {err_msg}",
|
|
||||||
chatbot=chatbot,
|
|
||||||
history=history,
|
|
||||||
delay=0
|
|
||||||
)
|
|
||||||
return
|
|
||||||
|
|
||||||
intention_type = pydantic_cls_instance.intention_type
|
|
||||||
intention_callback = self.tools_to_select[pydantic_cls_instance.intention_type]['callback']
|
|
||||||
yield from intention_callback(txt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, run_gpt_fn, intention_type)
|
|
||||||
|
|
||||||
|
|
||||||
def add_friend(self, friend):
|
|
||||||
# check whether the friend is already in the social network
|
|
||||||
for f in self.social_network.people:
|
|
||||||
if f.friend_name == friend.friend_name:
|
|
||||||
f.friend_description = friend.friend_description
|
|
||||||
f.friend_relationship = friend.friend_relationship
|
|
||||||
logger.info(f"Repeated friend, update info: {friend}")
|
|
||||||
return
|
|
||||||
logger.info(f"Add a new friend: {friend}")
|
|
||||||
self.social_network.people.append(friend)
|
|
||||||
return
|
|
||||||
|
|
||||||
|
|
||||||
@CatchException
|
|
||||||
def I人助手(txt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_request):
|
|
||||||
|
|
||||||
# 1. we retrieve worker from global context
|
|
||||||
user_name = chatbot.get_user()
|
|
||||||
checkpoint_dir=get_log_folder(user_name, plugin_name='experimental_rag')
|
|
||||||
if user_name in SOCIAL_NETWOK_WORKER_REGISTER:
|
|
||||||
social_network_worker = SOCIAL_NETWOK_WORKER_REGISTER[user_name]
|
|
||||||
else:
|
|
||||||
social_network_worker = SOCIAL_NETWOK_WORKER_REGISTER[user_name] = SocialNetworkWorker(
|
|
||||||
user_name,
|
|
||||||
llm_kwargs,
|
|
||||||
checkpoint_dir=checkpoint_dir,
|
|
||||||
auto_load_checkpoint=True
|
|
||||||
)
|
|
||||||
|
|
||||||
# 2. save
|
|
||||||
yield from social_network_worker.run(txt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_request)
|
|
||||||
social_network_worker.save_to_checkpoint(checkpoint_dir)
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
|
|
||||||
@@ -1,389 +0,0 @@
|
|||||||
from toolbox import update_ui, promote_file_to_downloadzone
|
|
||||||
from toolbox import CatchException, report_exception, write_history_to_file
|
|
||||||
from shared_utils.fastapi_server import validate_path_safety
|
|
||||||
from crazy_functions.crazy_utils import input_clipping
|
|
||||||
|
|
||||||
def 解析源代码新(file_manifest, project_folder, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt):
|
|
||||||
import os, copy
|
|
||||||
from crazy_functions.crazy_utils import request_gpt_model_multi_threads_with_very_awesome_ui_and_high_efficiency
|
|
||||||
from crazy_functions.crazy_utils import request_gpt_model_in_new_thread_with_ui_alive
|
|
||||||
|
|
||||||
summary_batch_isolation = True
|
|
||||||
inputs_array = []
|
|
||||||
inputs_show_user_array = []
|
|
||||||
history_array = []
|
|
||||||
sys_prompt_array = []
|
|
||||||
report_part_1 = []
|
|
||||||
|
|
||||||
assert len(file_manifest) <= 512, "源文件太多（超过512个）, 请缩减输入文件的数量。或者，您也可以选择删除此行警告，并修改代码拆分file_manifest列表，从而实现分批次处理。"
|
|
||||||
############################## <第一步，逐个文件分析，多线程> ##################################
|
|
||||||
for index, fp in enumerate(file_manifest):
|
|
||||||
# 读取文件
|
|
||||||
with open(fp, 'r', encoding='utf-8', errors='replace') as f:
|
|
||||||
file_content = f.read()
|
|
||||||
prefix = "接下来请你逐文件分析下面的工程" if index==0 else ""
|
|
||||||
i_say = prefix + f'请对下面的程序文件做一个概述文件名是{os.path.relpath(fp, project_folder)}，文件代码是 ```{file_content}```'
|
|
||||||
i_say_show_user = prefix + f'[{index+1}/{len(file_manifest)}] 请对下面的程序文件做一个概述: {fp}'
|
|
||||||
# 装载请求内容
|
|
||||||
inputs_array.append(i_say)
|
|
||||||
inputs_show_user_array.append(i_say_show_user)
|
|
||||||
history_array.append([])
|
|
||||||
sys_prompt_array.append("你是一个程序架构分析师，正在分析一个源代码项目。你的回答必须简单明了。")
|
|
||||||
|
|
||||||
# 文件读取完成，对每一个源代码文件，生成一个请求线程，发送到chatgpt进行分析
|
|
||||||
gpt_response_collection = yield from request_gpt_model_multi_threads_with_very_awesome_ui_and_high_efficiency(
|
|
||||||
inputs_array = inputs_array,
|
|
||||||
inputs_show_user_array = inputs_show_user_array,
|
|
||||||
history_array = history_array,
|
|
||||||
sys_prompt_array = sys_prompt_array,
|
|
||||||
llm_kwargs = llm_kwargs,
|
|
||||||
chatbot = chatbot,
|
|
||||||
show_user_at_complete = True
|
|
||||||
)
|
|
||||||
|
|
||||||
# 全部文件解析完成，结果写入文件，准备对工程源代码进行汇总分析
|
|
||||||
report_part_1 = copy.deepcopy(gpt_response_collection)
|
|
||||||
history_to_return = report_part_1
|
|
||||||
res = write_history_to_file(report_part_1)
|
|
||||||
promote_file_to_downloadzone(res, chatbot=chatbot)
|
|
||||||
chatbot.append(("完成？", "逐个文件分析已完成。" + res + "\n\n正在开始汇总。"))
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history_to_return) # 刷新界面
|
|
||||||
|
|
||||||
############################## <第二步，综合，单线程，分组+迭代处理> ##################################
|
|
||||||
batchsize = 16 # 10个文件为一组
|
|
||||||
report_part_2 = []
|
|
||||||
previous_iteration_files = []
|
|
||||||
last_iteration_result = ""
|
|
||||||
while True:
|
|
||||||
if len(file_manifest) == 0: break
|
|
||||||
this_iteration_file_manifest = file_manifest[:batchsize]
|
|
||||||
this_iteration_gpt_response_collection = gpt_response_collection[:batchsize*2]
|
|
||||||
file_rel_path = [os.path.relpath(fp, project_folder) for index, fp in enumerate(this_iteration_file_manifest)]
|
|
||||||
# 把“请对下面的程序文件做一个概述” 替换成 精简的 "文件名：{all_file[index]}"
|
|
||||||
for index, content in enumerate(this_iteration_gpt_response_collection):
|
|
||||||
if index%2==0: this_iteration_gpt_response_collection[index] = f"{file_rel_path[index//2]}" # 只保留文件名节省token
|
|
||||||
this_iteration_files = [os.path.relpath(fp, project_folder) for index, fp in enumerate(this_iteration_file_manifest)]
|
|
||||||
previous_iteration_files.extend(this_iteration_files)
|
|
||||||
previous_iteration_files_string = ', '.join(previous_iteration_files)
|
|
||||||
current_iteration_focus = ', '.join(this_iteration_files)
|
|
||||||
if summary_batch_isolation: focus = current_iteration_focus
|
|
||||||
else: focus = previous_iteration_files_string
|
|
||||||
i_say = f'用一张Markdown表格简要描述以下文件的功能：{focus}。根据以上分析，用一句话概括程序的整体功能。'
|
|
||||||
if last_iteration_result != "":
|
|
||||||
sys_prompt_additional = "已知某些代码的局部作用是:" + last_iteration_result + "\n请继续分析其他源代码，从而更全面地理解项目的整体功能。"
|
|
||||||
else:
|
|
||||||
sys_prompt_additional = ""
|
|
||||||
inputs_show_user = f'根据以上分析，对程序的整体功能和构架重新做出概括，由于输入长度限制，可能需要分组处理，本组文件为 {current_iteration_focus} + 已经汇总的文件组。'
|
|
||||||
this_iteration_history = copy.deepcopy(this_iteration_gpt_response_collection)
|
|
||||||
this_iteration_history.append(last_iteration_result)
|
|
||||||
# 裁剪input
|
|
||||||
inputs, this_iteration_history_feed = input_clipping(inputs=i_say, history=this_iteration_history, max_token_limit=2560)
|
|
||||||
result = yield from request_gpt_model_in_new_thread_with_ui_alive(
|
|
||||||
inputs=inputs, inputs_show_user=inputs_show_user, llm_kwargs=llm_kwargs, chatbot=chatbot,
|
|
||||||
history=this_iteration_history_feed, # 迭代之前的分析
|
|
||||||
sys_prompt="你是一个程序架构分析师，正在分析一个项目的源代码。" + sys_prompt_additional)
|
|
||||||
|
|
||||||
diagram_code = make_diagram(this_iteration_files, result, this_iteration_history_feed)
|
|
||||||
summary = "请用一句话概括这些文件的整体功能。\n\n" + diagram_code
|
|
||||||
summary_result = yield from request_gpt_model_in_new_thread_with_ui_alive(
|
|
||||||
inputs=summary,
|
|
||||||
inputs_show_user=summary,
|
|
||||||
llm_kwargs=llm_kwargs,
|
|
||||||
chatbot=chatbot,
|
|
||||||
history=[i_say, result], # 迭代之前的分析
|
|
||||||
sys_prompt="你是一个程序架构分析师，正在分析一个项目的源代码。" + sys_prompt_additional)
|
|
||||||
|
|
||||||
report_part_2.extend([i_say, result])
|
|
||||||
last_iteration_result = summary_result
|
|
||||||
file_manifest = file_manifest[batchsize:]
|
|
||||||
gpt_response_collection = gpt_response_collection[batchsize*2:]
|
|
||||||
|
|
||||||
############################## <END> ##################################
|
|
||||||
history_to_return.extend(report_part_2)
|
|
||||||
res = write_history_to_file(history_to_return)
|
|
||||||
promote_file_to_downloadzone(res, chatbot=chatbot)
|
|
||||||
chatbot.append(("完成了吗？", res))
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history_to_return) # 刷新界面
|
|
||||||
|
|
||||||
def make_diagram(this_iteration_files, result, this_iteration_history_feed):
|
|
||||||
from crazy_functions.diagram_fns.file_tree import build_file_tree_mermaid_diagram
|
|
||||||
return build_file_tree_mermaid_diagram(this_iteration_history_feed[0::2], this_iteration_history_feed[1::2], "项目示意图")
|
|
||||||
|
|
||||||
@CatchException
|
|
||||||
def 解析项目本身(txt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_request):
|
|
||||||
history = [] # 清空历史，以免输入溢出
|
|
||||||
import glob
|
|
||||||
file_manifest = [f for f in glob.glob('./*.py')] + \
|
|
||||||
[f for f in glob.glob('./*/*.py')]
|
|
||||||
project_folder = './'
|
|
||||||
if len(file_manifest) == 0:
|
|
||||||
report_exception(chatbot, history, a = f"解析项目: {txt}", b = f"找不到任何python文件: {txt}")
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
yield from 解析源代码新(file_manifest, project_folder, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt)
|
|
||||||
|
|
||||||
@CatchException
|
|
||||||
def 解析一个Python项目(txt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_request):
|
|
||||||
history = [] # 清空历史，以免输入溢出
|
|
||||||
import glob, os
|
|
||||||
if os.path.exists(txt):
|
|
||||||
project_folder = txt
|
|
||||||
validate_path_safety(project_folder, chatbot.get_user())
|
|
||||||
else:
|
|
||||||
if txt == "": txt = '空空如也的输入栏'
|
|
||||||
report_exception(chatbot, history, a = f"解析项目: {txt}", b = f"找不到本地项目或无权访问: {txt}")
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
file_manifest = [f for f in glob.glob(f'{project_folder}/**/*.py', recursive=True)]
|
|
||||||
if len(file_manifest) == 0:
|
|
||||||
report_exception(chatbot, history, a = f"解析项目: {txt}", b = f"找不到任何python文件: {txt}")
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
yield from 解析源代码新(file_manifest, project_folder, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt)
|
|
||||||
|
|
||||||
@CatchException
|
|
||||||
def 解析一个Matlab项目(txt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_request):
|
|
||||||
history = [] # 清空历史，以免输入溢出
|
|
||||||
import glob, os
|
|
||||||
if os.path.exists(txt):
|
|
||||||
project_folder = txt
|
|
||||||
validate_path_safety(project_folder, chatbot.get_user())
|
|
||||||
else:
|
|
||||||
if txt == "": txt = '空空如也的输入栏'
|
|
||||||
report_exception(chatbot, history, a = f"解析Matlab项目: {txt}", b = f"找不到本地项目或无权访问: {txt}")
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
file_manifest = [f for f in glob.glob(f'{project_folder}/**/*.m', recursive=True)]
|
|
||||||
if len(file_manifest) == 0:
|
|
||||||
report_exception(chatbot, history, a = f"解析Matlab项目: {txt}", b = f"找不到任何`.m`源文件: {txt}")
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
yield from 解析源代码新(file_manifest, project_folder, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt)
|
|
||||||
|
|
||||||
@CatchException
|
|
||||||
def 解析一个C项目的头文件(txt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_request):
|
|
||||||
history = [] # 清空历史，以免输入溢出
|
|
||||||
import glob, os
|
|
||||||
if os.path.exists(txt):
|
|
||||||
project_folder = txt
|
|
||||||
validate_path_safety(project_folder, chatbot.get_user())
|
|
||||||
else:
|
|
||||||
if txt == "": txt = '空空如也的输入栏'
|
|
||||||
report_exception(chatbot, history, a = f"解析项目: {txt}", b = f"找不到本地项目或无权访问: {txt}")
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
file_manifest = [f for f in glob.glob(f'{project_folder}/**/*.h', recursive=True)] + \
|
|
||||||
[f for f in glob.glob(f'{project_folder}/**/*.hpp', recursive=True)] #+ \
|
|
||||||
# [f for f in glob.glob(f'{project_folder}/**/*.c', recursive=True)]
|
|
||||||
if len(file_manifest) == 0:
|
|
||||||
report_exception(chatbot, history, a = f"解析项目: {txt}", b = f"找不到任何.h头文件: {txt}")
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
yield from 解析源代码新(file_manifest, project_folder, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt)
|
|
||||||
|
|
||||||
@CatchException
|
|
||||||
def 解析一个C项目(txt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_request):
|
|
||||||
history = [] # 清空历史，以免输入溢出
|
|
||||||
import glob, os
|
|
||||||
if os.path.exists(txt):
|
|
||||||
project_folder = txt
|
|
||||||
validate_path_safety(project_folder, chatbot.get_user())
|
|
||||||
else:
|
|
||||||
if txt == "": txt = '空空如也的输入栏'
|
|
||||||
report_exception(chatbot, history, a = f"解析项目: {txt}", b = f"找不到本地项目或无权访问: {txt}")
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
file_manifest = [f for f in glob.glob(f'{project_folder}/**/*.h', recursive=True)] + \
|
|
||||||
[f for f in glob.glob(f'{project_folder}/**/*.cpp', recursive=True)] + \
|
|
||||||
[f for f in glob.glob(f'{project_folder}/**/*.hpp', recursive=True)] + \
|
|
||||||
[f for f in glob.glob(f'{project_folder}/**/*.c', recursive=True)]
|
|
||||||
if len(file_manifest) == 0:
|
|
||||||
report_exception(chatbot, history, a = f"解析项目: {txt}", b = f"找不到任何.h头文件: {txt}")
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
yield from 解析源代码新(file_manifest, project_folder, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt)
|
|
||||||
|
|
||||||
|
|
||||||
@CatchException
|
|
||||||
def 解析一个Java项目(txt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_request):
|
|
||||||
history = [] # 清空历史，以免输入溢出
|
|
||||||
import glob, os
|
|
||||||
if os.path.exists(txt):
|
|
||||||
project_folder = txt
|
|
||||||
validate_path_safety(project_folder, chatbot.get_user())
|
|
||||||
else:
|
|
||||||
if txt == "": txt = '空空如也的输入栏'
|
|
||||||
report_exception(chatbot, history, a=f"解析项目: {txt}", b=f"找不到本地项目或无权访问: {txt}")
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
file_manifest = [f for f in glob.glob(f'{project_folder}/**/*.java', recursive=True)] + \
|
|
||||||
[f for f in glob.glob(f'{project_folder}/**/*.jar', recursive=True)] + \
|
|
||||||
[f for f in glob.glob(f'{project_folder}/**/*.xml', recursive=True)] + \
|
|
||||||
[f for f in glob.glob(f'{project_folder}/**/*.sh', recursive=True)]
|
|
||||||
if len(file_manifest) == 0:
|
|
||||||
report_exception(chatbot, history, a=f"解析项目: {txt}", b=f"找不到任何java文件: {txt}")
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
yield from 解析源代码新(file_manifest, project_folder, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt)
|
|
||||||
|
|
||||||
|
|
||||||
@CatchException
|
|
||||||
def 解析一个前端项目(txt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_request):
|
|
||||||
history = [] # 清空历史，以免输入溢出
|
|
||||||
import glob, os
|
|
||||||
if os.path.exists(txt):
|
|
||||||
project_folder = txt
|
|
||||||
validate_path_safety(project_folder, chatbot.get_user())
|
|
||||||
else:
|
|
||||||
if txt == "": txt = '空空如也的输入栏'
|
|
||||||
report_exception(chatbot, history, a=f"解析项目: {txt}", b=f"找不到本地项目或无权访问: {txt}")
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
file_manifest = [f for f in glob.glob(f'{project_folder}/**/*.ts', recursive=True)] + \
|
|
||||||
[f for f in glob.glob(f'{project_folder}/**/*.tsx', recursive=True)] + \
|
|
||||||
[f for f in glob.glob(f'{project_folder}/**/*.json', recursive=True)] + \
|
|
||||||
[f for f in glob.glob(f'{project_folder}/**/*.js', recursive=True)] + \
|
|
||||||
[f for f in glob.glob(f'{project_folder}/**/*.vue', recursive=True)] + \
|
|
||||||
[f for f in glob.glob(f'{project_folder}/**/*.less', recursive=True)] + \
|
|
||||||
[f for f in glob.glob(f'{project_folder}/**/*.sass', recursive=True)] + \
|
|
||||||
[f for f in glob.glob(f'{project_folder}/**/*.wxml', recursive=True)] + \
|
|
||||||
[f for f in glob.glob(f'{project_folder}/**/*.wxss', recursive=True)] + \
|
|
||||||
[f for f in glob.glob(f'{project_folder}/**/*.css', recursive=True)] + \
|
|
||||||
[f for f in glob.glob(f'{project_folder}/**/*.jsx', recursive=True)]
|
|
||||||
if len(file_manifest) == 0:
|
|
||||||
report_exception(chatbot, history, a=f"解析项目: {txt}", b=f"找不到任何前端相关文件: {txt}")
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
yield from 解析源代码新(file_manifest, project_folder, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt)
|
|
||||||
|
|
||||||
|
|
||||||
@CatchException
|
|
||||||
def 解析一个Golang项目(txt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_request):
|
|
||||||
history = [] # 清空历史，以免输入溢出
|
|
||||||
import glob, os
|
|
||||||
if os.path.exists(txt):
|
|
||||||
project_folder = txt
|
|
||||||
validate_path_safety(project_folder, chatbot.get_user())
|
|
||||||
else:
|
|
||||||
if txt == "": txt = '空空如也的输入栏'
|
|
||||||
report_exception(chatbot, history, a=f"解析项目: {txt}", b=f"找不到本地项目或无权访问: {txt}")
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
file_manifest = [f for f in glob.glob(f'{project_folder}/**/*.go', recursive=True)] + \
|
|
||||||
[f for f in glob.glob(f'{project_folder}/**/go.mod', recursive=True)] + \
|
|
||||||
[f for f in glob.glob(f'{project_folder}/**/go.sum', recursive=True)] + \
|
|
||||||
[f for f in glob.glob(f'{project_folder}/**/go.work', recursive=True)]
|
|
||||||
if len(file_manifest) == 0:
|
|
||||||
report_exception(chatbot, history, a=f"解析项目: {txt}", b=f"找不到任何golang文件: {txt}")
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
yield from 解析源代码新(file_manifest, project_folder, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt)
|
|
||||||
|
|
||||||
@CatchException
|
|
||||||
def 解析一个Rust项目(txt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_request):
|
|
||||||
history = [] # 清空历史，以免输入溢出
|
|
||||||
import glob, os
|
|
||||||
if os.path.exists(txt):
|
|
||||||
project_folder = txt
|
|
||||||
validate_path_safety(project_folder, chatbot.get_user())
|
|
||||||
else:
|
|
||||||
if txt == "": txt = '空空如也的输入栏'
|
|
||||||
report_exception(chatbot, history, a=f"解析项目: {txt}", b=f"找不到本地项目或无权访问: {txt}")
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
file_manifest = [f for f in glob.glob(f'{project_folder}/**/*.rs', recursive=True)] + \
|
|
||||||
[f for f in glob.glob(f'{project_folder}/**/*.toml', recursive=True)] + \
|
|
||||||
[f for f in glob.glob(f'{project_folder}/**/*.lock', recursive=True)]
|
|
||||||
if len(file_manifest) == 0:
|
|
||||||
report_exception(chatbot, history, a=f"解析项目: {txt}", b=f"找不到任何golang文件: {txt}")
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
yield from 解析源代码新(file_manifest, project_folder, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt)
|
|
||||||
|
|
||||||
@CatchException
|
|
||||||
def 解析一个Lua项目(txt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_request):
|
|
||||||
history = [] # 清空历史，以免输入溢出
|
|
||||||
import glob, os
|
|
||||||
if os.path.exists(txt):
|
|
||||||
project_folder = txt
|
|
||||||
validate_path_safety(project_folder, chatbot.get_user())
|
|
||||||
else:
|
|
||||||
if txt == "": txt = '空空如也的输入栏'
|
|
||||||
report_exception(chatbot, history, a = f"解析项目: {txt}", b = f"找不到本地项目或无权访问: {txt}")
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
file_manifest = [f for f in glob.glob(f'{project_folder}/**/*.lua', recursive=True)] + \
|
|
||||||
[f for f in glob.glob(f'{project_folder}/**/*.xml', recursive=True)] + \
|
|
||||||
[f for f in glob.glob(f'{project_folder}/**/*.json', recursive=True)] + \
|
|
||||||
[f for f in glob.glob(f'{project_folder}/**/*.toml', recursive=True)]
|
|
||||||
if len(file_manifest) == 0:
|
|
||||||
report_exception(chatbot, history, a = f"解析项目: {txt}", b = f"找不到任何lua文件: {txt}")
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
yield from 解析源代码新(file_manifest, project_folder, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt)
|
|
||||||
|
|
||||||
|
|
||||||
@CatchException
|
|
||||||
def 解析一个CSharp项目(txt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_request):
|
|
||||||
history = [] # 清空历史，以免输入溢出
|
|
||||||
import glob, os
|
|
||||||
if os.path.exists(txt):
|
|
||||||
project_folder = txt
|
|
||||||
validate_path_safety(project_folder, chatbot.get_user())
|
|
||||||
else:
|
|
||||||
if txt == "": txt = '空空如也的输入栏'
|
|
||||||
report_exception(chatbot, history, a = f"解析项目: {txt}", b = f"找不到本地项目或无权访问: {txt}")
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
file_manifest = [f for f in glob.glob(f'{project_folder}/**/*.cs', recursive=True)] + \
|
|
||||||
[f for f in glob.glob(f'{project_folder}/**/*.csproj', recursive=True)]
|
|
||||||
if len(file_manifest) == 0:
|
|
||||||
report_exception(chatbot, history, a = f"解析项目: {txt}", b = f"找不到任何CSharp文件: {txt}")
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
yield from 解析源代码新(file_manifest, project_folder, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt)
|
|
||||||
|
|
||||||
|
|
||||||
@CatchException
|
|
||||||
def 解析任意code项目(txt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_request):
|
|
||||||
txt_pattern = plugin_kwargs.get("advanced_arg")
|
|
||||||
txt_pattern = txt_pattern.replace("，", ",")
|
|
||||||
# 将要匹配的模式(例如: *.c, *.cpp, *.py, config.toml)
|
|
||||||
pattern_include = [_.lstrip(" ,").rstrip(" ,") for _ in txt_pattern.split(",") if _ != "" and not _.strip().startswith("^")]
|
|
||||||
if not pattern_include: pattern_include = ["*"] # 不输入即全部匹配
|
|
||||||
# 将要忽略匹配的文件后缀(例如: ^*.c, ^*.cpp, ^*.py)
|
|
||||||
pattern_except_suffix = [_.lstrip(" ^*.,").rstrip(" ,") for _ in txt_pattern.split(" ") if _ != "" and _.strip().startswith("^*.")]
|
|
||||||
pattern_except_suffix += ['zip', 'rar', '7z', 'tar', 'gz'] # 避免解析压缩文件
|
|
||||||
# 将要忽略匹配的文件名(例如: ^README.md)
|
|
||||||
pattern_except_name = [_.lstrip(" ^*,").rstrip(" ,").replace(".", r"\.") # 移除左边通配符，移除右侧逗号，转义点号
|
|
||||||
for _ in txt_pattern.split(" ") # 以空格分割
|
|
||||||
if (_ != "" and _.strip().startswith("^") and not _.strip().startswith("^*.")) # ^开始，但不是^*.开始
|
|
||||||
]
|
|
||||||
# 生成正则表达式
|
|
||||||
pattern_except = r'/[^/]+\.(' + "|".join(pattern_except_suffix) + ')$'
|
|
||||||
pattern_except += '|/(' + "|".join(pattern_except_name) + ')$' if pattern_except_name != [] else ''
|
|
||||||
|
|
||||||
history.clear()
|
|
||||||
import glob, os, re
|
|
||||||
if os.path.exists(txt):
|
|
||||||
project_folder = txt
|
|
||||||
validate_path_safety(project_folder, chatbot.get_user())
|
|
||||||
else:
|
|
||||||
if txt == "": txt = '空空如也的输入栏'
|
|
||||||
report_exception(chatbot, history, a = f"解析项目: {txt}", b = f"找不到本地项目或无权访问: {txt}")
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
# 若上传压缩文件, 先寻找到解压的文件夹路径, 从而避免解析压缩文件
|
|
||||||
maybe_dir = [f for f in glob.glob(f'{project_folder}/*') if os.path.isdir(f)]
|
|
||||||
if len(maybe_dir)>0 and maybe_dir[0].endswith('.extract'):
|
|
||||||
extract_folder_path = maybe_dir[0]
|
|
||||||
else:
|
|
||||||
extract_folder_path = project_folder
|
|
||||||
# 按输入的匹配模式寻找上传的非压缩文件和已解压的文件
|
|
||||||
file_manifest = [f for pattern in pattern_include for f in glob.glob(f'{extract_folder_path}/**/{pattern}', recursive=True) if "" != extract_folder_path and \
|
|
||||||
os.path.isfile(f) and (not re.search(pattern_except, f) or pattern.endswith('.' + re.search(pattern_except, f).group().split('.')[-1]))]
|
|
||||||
if len(file_manifest) == 0:
|
|
||||||
report_exception(chatbot, history, a = f"解析项目: {txt}", b = f"找不到任何文件: {txt}")
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
yield from 解析源代码新(file_manifest, project_folder, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt)
|
|
||||||
@@ -1,162 +0,0 @@
|
|||||||
import os, copy, time
|
|
||||||
from toolbox import CatchException, report_exception, update_ui, zip_result, promote_file_to_downloadzone, update_ui_lastest_msg, get_conf, generate_file_link
|
|
||||||
from shared_utils.fastapi_server import validate_path_safety
|
|
||||||
from crazy_functions.crazy_utils import input_clipping
|
|
||||||
from crazy_functions.crazy_utils import request_gpt_model_multi_threads_with_very_awesome_ui_and_high_efficiency
|
|
||||||
from crazy_functions.crazy_utils import request_gpt_model_in_new_thread_with_ui_alive
|
|
||||||
from crazy_functions.agent_fns.python_comment_agent import PythonCodeComment
|
|
||||||
from crazy_functions.diagram_fns.file_tree import FileNode
|
|
||||||
from crazy_functions.agent_fns.watchdog import WatchDog
|
|
||||||
from shared_utils.advanced_markdown_format import markdown_convertion_for_file
|
|
||||||
from loguru import logger
|
|
||||||
|
|
||||||
|
|
||||||
def 注释源代码(file_manifest, project_folder, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt):
|
|
||||||
|
|
||||||
summary_batch_isolation = True
|
|
||||||
inputs_array = []
|
|
||||||
inputs_show_user_array = []
|
|
||||||
history_array = []
|
|
||||||
sys_prompt_array = []
|
|
||||||
|
|
||||||
assert len(file_manifest) <= 512, "源文件太多（超过512个）, 请缩减输入文件的数量。或者，您也可以选择删除此行警告，并修改代码拆分file_manifest列表，从而实现分批次处理。"
|
|
||||||
|
|
||||||
# 建立文件树
|
|
||||||
file_tree_struct = FileNode("root", build_manifest=True)
|
|
||||||
for file_path in file_manifest:
|
|
||||||
file_tree_struct.add_file(file_path, file_path)
|
|
||||||
|
|
||||||
# <第一步，逐个文件分析，多线程>
|
|
||||||
lang = "" if not plugin_kwargs["use_chinese"] else " (you must use Chinese)"
|
|
||||||
for index, fp in enumerate(file_manifest):
|
|
||||||
# 读取文件
|
|
||||||
with open(fp, 'r', encoding='utf-8', errors='replace') as f:
|
|
||||||
file_content = f.read()
|
|
||||||
prefix = ""
|
|
||||||
i_say = prefix + f'Please conclude the following source code at {os.path.relpath(fp, project_folder)} with only one sentence{lang}, the code is:\n```{file_content}```'
|
|
||||||
i_say_show_user = prefix + f'[{index+1}/{len(file_manifest)}] 请用一句话对下面的程序文件做一个整体概述: {fp}'
|
|
||||||
# 装载请求内容
|
|
||||||
MAX_TOKEN_SINGLE_FILE = 2560
|
|
||||||
i_say, _ = input_clipping(inputs=i_say, history=[], max_token_limit=MAX_TOKEN_SINGLE_FILE)
|
|
||||||
inputs_array.append(i_say)
|
|
||||||
inputs_show_user_array.append(i_say_show_user)
|
|
||||||
history_array.append([])
|
|
||||||
sys_prompt_array.append(f"You are a software architecture analyst analyzing a source code project. Do not dig into details, tell me what the code is doing in general. Your answer must be short, simple and clear{lang}.")
|
|
||||||
# 文件读取完成，对每一个源代码文件，生成一个请求线程，发送到大模型进行分析
|
|
||||||
gpt_response_collection = yield from request_gpt_model_multi_threads_with_very_awesome_ui_and_high_efficiency(
|
|
||||||
inputs_array = inputs_array,
|
|
||||||
inputs_show_user_array = inputs_show_user_array,
|
|
||||||
history_array = history_array,
|
|
||||||
sys_prompt_array = sys_prompt_array,
|
|
||||||
llm_kwargs = llm_kwargs,
|
|
||||||
chatbot = chatbot,
|
|
||||||
show_user_at_complete = True
|
|
||||||
)
|
|
||||||
|
|
||||||
# <第二步，逐个文件分析，生成带注释文件>
|
|
||||||
tasks = ["" for _ in range(len(file_manifest))]
|
|
||||||
def bark_fn(tasks):
|
|
||||||
for i in range(len(tasks)): tasks[i] = "watchdog is dead"
|
|
||||||
wd = WatchDog(timeout=10, bark_fn=lambda: bark_fn(tasks), interval=3, msg="ThreadWatcher timeout")
|
|
||||||
wd.begin_watch()
|
|
||||||
from concurrent.futures import ThreadPoolExecutor
|
|
||||||
executor = ThreadPoolExecutor(max_workers=get_conf('DEFAULT_WORKER_NUM'))
|
|
||||||
def _task_multi_threading(i_say, gpt_say, fp, file_tree_struct, index):
|
|
||||||
language = 'Chinese' if plugin_kwargs["use_chinese"] else 'English'
|
|
||||||
def observe_window_update(x):
|
|
||||||
if tasks[index] == "watchdog is dead":
|
|
||||||
raise TimeoutError("ThreadWatcher: watchdog is dead")
|
|
||||||
tasks[index] = x
|
|
||||||
pcc = PythonCodeComment(llm_kwargs, plugin_kwargs, language=language, observe_window_update=observe_window_update)
|
|
||||||
pcc.read_file(path=fp, brief=gpt_say)
|
|
||||||
revised_path, revised_content = pcc.begin_comment_source_code(None, None)
|
|
||||||
file_tree_struct.manifest[fp].revised_path = revised_path
|
|
||||||
file_tree_struct.manifest[fp].revised_content = revised_content
|
|
||||||
# <将结果写回源文件>
|
|
||||||
with open(fp, 'w', encoding='utf-8') as f:
|
|
||||||
f.write(file_tree_struct.manifest[fp].revised_content)
|
|
||||||
# <生成对比html>
|
|
||||||
with open("crazy_functions/agent_fns/python_comment_compare.html", 'r', encoding='utf-8') as f:
|
|
||||||
html_template = f.read()
|
|
||||||
warp = lambda x: "```python\n\n" + x + "\n\n```"
|
|
||||||
from themes.theme import load_dynamic_theme
|
|
||||||
_, advanced_css, _, _ = load_dynamic_theme("Default")
|
|
||||||
html_template = html_template.replace("ADVANCED_CSS", advanced_css)
|
|
||||||
html_template = html_template.replace("REPLACE_CODE_FILE_LEFT", pcc.get_markdown_block_in_html(markdown_convertion_for_file(warp(pcc.original_content))))
|
|
||||||
html_template = html_template.replace("REPLACE_CODE_FILE_RIGHT", pcc.get_markdown_block_in_html(markdown_convertion_for_file(warp(revised_content))))
|
|
||||||
compare_html_path = fp + '.compare.html'
|
|
||||||
file_tree_struct.manifest[fp].compare_html = compare_html_path
|
|
||||||
with open(compare_html_path, 'w', encoding='utf-8') as f:
|
|
||||||
f.write(html_template)
|
|
||||||
tasks[index] = ""
|
|
||||||
|
|
||||||
chatbot.append([None, f"正在处理:"])
|
|
||||||
futures = []
|
|
||||||
index = 0
|
|
||||||
for i_say, gpt_say, fp in zip(gpt_response_collection[0::2], gpt_response_collection[1::2], file_manifest):
|
|
||||||
future = executor.submit(_task_multi_threading, i_say, gpt_say, fp, file_tree_struct, index)
|
|
||||||
index += 1
|
|
||||||
futures.append(future)
|
|
||||||
|
|
||||||
# <第三步，等待任务完成>
|
|
||||||
cnt = 0
|
|
||||||
while True:
|
|
||||||
cnt += 1
|
|
||||||
wd.feed()
|
|
||||||
time.sleep(3)
|
|
||||||
worker_done = [h.done() for h in futures]
|
|
||||||
remain = len(worker_done) - sum(worker_done)
|
|
||||||
|
|
||||||
# <展示已经完成的部分>
|
|
||||||
preview_html_list = []
|
|
||||||
for done, fp in zip(worker_done, file_manifest):
|
|
||||||
if not done: continue
|
|
||||||
if hasattr(file_tree_struct.manifest[fp], 'compare_html'):
|
|
||||||
preview_html_list.append(file_tree_struct.manifest[fp].compare_html)
|
|
||||||
else:
|
|
||||||
logger.error(f"文件: {fp} 的注释结果未能成功")
|
|
||||||
file_links = generate_file_link(preview_html_list)
|
|
||||||
|
|
||||||
yield from update_ui_lastest_msg(
|
|
||||||
f"当前任务: <br/>{'<br/>'.join(tasks)}.<br/>" +
|
|
||||||
f"剩余源文件数量: {remain}.<br/>" +
|
|
||||||
f"已完成的文件: {sum(worker_done)}.<br/>" +
|
|
||||||
file_links +
|
|
||||||
"<br/>" +
|
|
||||||
''.join(['.']*(cnt % 10 + 1)
|
|
||||||
), chatbot=chatbot, history=history, delay=0)
|
|
||||||
yield from update_ui(chatbot=chatbot, history=[]) # 刷新界面
|
|
||||||
if all(worker_done):
|
|
||||||
executor.shutdown()
|
|
||||||
break
|
|
||||||
|
|
||||||
# <第四步，压缩结果>
|
|
||||||
zip_res = zip_result(project_folder)
|
|
||||||
promote_file_to_downloadzone(file=zip_res, chatbot=chatbot)
|
|
||||||
|
|
||||||
# <END>
|
|
||||||
chatbot.append((None, "所有源文件均已处理完毕。"))
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@CatchException
|
|
||||||
def 注释Python项目(txt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_request):
|
|
||||||
history = [] # 清空历史，以免输入溢出
|
|
||||||
plugin_kwargs["use_chinese"] = plugin_kwargs.get("use_chinese", False)
|
|
||||||
import glob, os
|
|
||||||
if os.path.exists(txt):
|
|
||||||
project_folder = txt
|
|
||||||
validate_path_safety(project_folder, chatbot.get_user())
|
|
||||||
else:
|
|
||||||
if txt == "": txt = '空空如也的输入栏'
|
|
||||||
report_exception(chatbot, history, a = f"解析项目: {txt}", b = f"找不到本地项目或无权访问: {txt}")
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
file_manifest = [f for f in glob.glob(f'{project_folder}/**/*.py', recursive=True)]
|
|
||||||
if len(file_manifest) == 0:
|
|
||||||
report_exception(chatbot, history, a = f"解析项目: {txt}", b = f"找不到任何python文件: {txt}")
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
|
|
||||||
yield from 注释源代码(file_manifest, project_folder, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt)
|
|
||||||
@@ -1,36 +0,0 @@
|
|||||||
|
|
||||||
from toolbox import get_conf, update_ui
|
|
||||||
from crazy_functions.plugin_template.plugin_class_template import GptAcademicPluginTemplate, ArgProperty
|
|
||||||
from crazy_functions.SourceCode_Comment import 注释Python项目
|
|
||||||
|
|
||||||
class SourceCodeComment_Wrap(GptAcademicPluginTemplate):
|
|
||||||
def __init__(self):
|
|
||||||
"""
|
|
||||||
请注意`execute`会执行在不同的线程中，因此您在定义和使用类变量时，应当慎之又慎！
|
|
||||||
"""
|
|
||||||
pass
|
|
||||||
|
|
||||||
def define_arg_selection_menu(self):
|
|
||||||
"""
|
|
||||||
定义插件的二级选项菜单
|
|
||||||
"""
|
|
||||||
gui_definition = {
|
|
||||||
"main_input":
|
|
||||||
ArgProperty(title="路径", description="程序路径（上传文件后自动填写）", default_value="", type="string").model_dump_json(), # 主输入，自动从输入框同步
|
|
||||||
"use_chinese":
|
|
||||||
ArgProperty(title="注释语言", options=["英文", "中文"], default_value="英文", description="无", type="dropdown").model_dump_json(),
|
|
||||||
# "use_emoji":
|
|
||||||
# ArgProperty(title="在注释中使用emoji", options=["禁止", "允许"], default_value="禁止", description="无", type="dropdown").model_dump_json(),
|
|
||||||
}
|
|
||||||
return gui_definition
|
|
||||||
|
|
||||||
def execute(txt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_request):
|
|
||||||
"""
|
|
||||||
执行插件
|
|
||||||
"""
|
|
||||||
if plugin_kwargs["use_chinese"] == "中文":
|
|
||||||
plugin_kwargs["use_chinese"] = True
|
|
||||||
else:
|
|
||||||
plugin_kwargs["use_chinese"] = False
|
|
||||||
|
|
||||||
yield from 注释Python项目(txt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_request)
|
|
||||||
@@ -1,204 +0,0 @@
|
|||||||
import requests
|
|
||||||
import random
|
|
||||||
import time
|
|
||||||
import re
|
|
||||||
import json
|
|
||||||
from bs4 import BeautifulSoup
|
|
||||||
from functools import lru_cache
|
|
||||||
from itertools import zip_longest
|
|
||||||
from check_proxy import check_proxy
|
|
||||||
from toolbox import CatchException, update_ui, get_conf, promote_file_to_downloadzone, update_ui_lastest_msg, generate_file_link
|
|
||||||
from crazy_functions.crazy_utils import request_gpt_model_in_new_thread_with_ui_alive, input_clipping
|
|
||||||
from request_llms.bridge_all import model_info
|
|
||||||
from request_llms.bridge_all import predict_no_ui_long_connection
|
|
||||||
from crazy_functions.prompts.internet import SearchOptimizerPrompt, SearchAcademicOptimizerPrompt
|
|
||||||
from crazy_functions.json_fns.pydantic_io import GptJsonIO, JsonStringError
|
|
||||||
from textwrap import dedent
|
|
||||||
from loguru import logger
|
|
||||||
from pydantic import BaseModel, Field
|
|
||||||
|
|
||||||
class Query(BaseModel):
|
|
||||||
search_keyword: str = Field(description="search query for video resource")
|
|
||||||
|
|
||||||
|
|
||||||
class VideoResource(BaseModel):
|
|
||||||
thought: str = Field(description="analysis of the search results based on the user's query")
|
|
||||||
title: str = Field(description="title of the video")
|
|
||||||
author: str = Field(description="author/uploader of the video")
|
|
||||||
bvid: str = Field(description="unique ID of the video")
|
|
||||||
another_failsafe_bvid: str = Field(description="provide another bvid, the other one is not working")
|
|
||||||
|
|
||||||
|
|
||||||
def get_video_resource(search_keyword):
|
|
||||||
from crazy_functions.media_fns.get_media import search_videos
|
|
||||||
|
|
||||||
# Search for videos and return the first result
|
|
||||||
videos = search_videos(
|
|
||||||
search_keyword
|
|
||||||
)
|
|
||||||
|
|
||||||
# Return the first video if results exist, otherwise return None
|
|
||||||
return videos
|
|
||||||
|
|
||||||
def download_video(bvid, user_name, chatbot, history):
|
|
||||||
# from experimental_mods.get_bilibili_resource import download_bilibili
|
|
||||||
from crazy_functions.media_fns.get_media import download_video
|
|
||||||
# pause a while
|
|
||||||
tic_time = 8
|
|
||||||
for i in range(tic_time):
|
|
||||||
yield from update_ui_lastest_msg(
|
|
||||||
lastmsg=f"即将下载音频。等待{tic_time-i}秒后自动继续, 点击“停止”键取消此操作。",
|
|
||||||
chatbot=chatbot, history=[], delay=1)
|
|
||||||
|
|
||||||
# download audio
|
|
||||||
chatbot.append((None, "下载音频, 请稍等...")); yield from update_ui(chatbot=chatbot, history=history)
|
|
||||||
downloaded_files = yield from download_video(bvid, only_audio=True, user_name=user_name, chatbot=chatbot, history=history)
|
|
||||||
|
|
||||||
if len(downloaded_files) == 0:
|
|
||||||
# failed to download audio
|
|
||||||
return []
|
|
||||||
|
|
||||||
# preview
|
|
||||||
preview_list = [promote_file_to_downloadzone(fp) for fp in downloaded_files]
|
|
||||||
file_links = generate_file_link(preview_list)
|
|
||||||
yield from update_ui_lastest_msg(f"已完成的文件: <br/>" + file_links, chatbot=chatbot, history=history, delay=0)
|
|
||||||
chatbot.append((None, f"即将下载视频。"))
|
|
||||||
|
|
||||||
# pause a while
|
|
||||||
tic_time = 16
|
|
||||||
for i in range(tic_time):
|
|
||||||
yield from update_ui_lastest_msg(
|
|
||||||
lastmsg=f"即将下载视频。等待{tic_time-i}秒后自动继续, 点击“停止”键取消此操作。",
|
|
||||||
chatbot=chatbot, history=[], delay=1)
|
|
||||||
|
|
||||||
# download video
|
|
||||||
chatbot.append((None, "下载视频, 请稍等...")); yield from update_ui(chatbot=chatbot, history=history)
|
|
||||||
downloaded_files_part2 = yield from download_video(bvid, only_audio=False, user_name=user_name, chatbot=chatbot, history=history)
|
|
||||||
|
|
||||||
# preview
|
|
||||||
preview_list = [promote_file_to_downloadzone(fp) for fp in downloaded_files_part2]
|
|
||||||
file_links = generate_file_link(preview_list)
|
|
||||||
yield from update_ui_lastest_msg(f"已完成的文件: <br/>" + file_links, chatbot=chatbot, history=history, delay=0)
|
|
||||||
|
|
||||||
# return
|
|
||||||
return downloaded_files + downloaded_files_part2
|
|
||||||
|
|
||||||
|
|
||||||
class Strategy(BaseModel):
|
|
||||||
thought: str = Field(description="analysis of the user's wish, for example, can you recall the name of the resource?")
|
|
||||||
which_methods: str = Field(description="Which method to use to find the necessary information? choose from 'method_1' and 'method_2'.")
|
|
||||||
method_1_search_keywords: str = Field(description="Generate keywords to search the internet if you choose method 1, otherwise empty.")
|
|
||||||
method_2_generate_keywords: str = Field(description="Generate keywords for video download engine if you choose method 2, otherwise empty.")
|
|
||||||
|
|
||||||
|
|
||||||
@CatchException
|
|
||||||
def 多媒体任务(txt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_request):
|
|
||||||
user_wish: str = txt
|
|
||||||
# query demos:
|
|
||||||
# - "我想找一首歌，里面有句歌词是“turn your face towards the sun”"
|
|
||||||
# - "一首歌，第一句是红豆生南国"
|
|
||||||
# - "一首音乐，中国航天任务专用的那首"
|
|
||||||
# - "戴森球计划在熔岩星球的音乐"
|
|
||||||
# - "hanser的百变什么精"
|
|
||||||
# - "打大圣残躯时的bgm"
|
|
||||||
# - "渊下宫战斗音乐"
|
|
||||||
|
|
||||||
# 搜索
|
|
||||||
chatbot.append((txt, "检索中, 请稍等..."))
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
if "跳过联网搜索" not in user_wish:
|
|
||||||
# 结构化生成
|
|
||||||
internet_search_keyword = user_wish
|
|
||||||
|
|
||||||
yield from update_ui_lastest_msg(lastmsg=f"发起互联网检索: {internet_search_keyword} ...", chatbot=chatbot, history=[], delay=1)
|
|
||||||
from crazy_functions.Internet_GPT import internet_search_with_analysis_prompt
|
|
||||||
result = yield from internet_search_with_analysis_prompt(
|
|
||||||
prompt=internet_search_keyword,
|
|
||||||
analysis_prompt="请根据搜索结果分析，获取用户需要找的资源的名称、作者、出处等信息。",
|
|
||||||
llm_kwargs=llm_kwargs,
|
|
||||||
chatbot=chatbot
|
|
||||||
)
|
|
||||||
|
|
||||||
yield from update_ui_lastest_msg(lastmsg=f"互联网检索结论: {result} \n\n 正在生成进一步检索方案 ...", chatbot=chatbot, history=[], delay=1)
|
|
||||||
rf_req = dedent(f"""
|
|
||||||
The user wish to get the following resource:
|
|
||||||
{user_wish}
|
|
||||||
Meanwhile, you can access another expert's opinion on the user's wish:
|
|
||||||
{result}
|
|
||||||
Generate search keywords (less than 5 keywords) for video download engine accordingly.
|
|
||||||
""")
|
|
||||||
else:
|
|
||||||
user_wish = user_wish.replace("跳过联网搜索", "").strip()
|
|
||||||
rf_req = dedent(f"""
|
|
||||||
The user wish to get the following resource:
|
|
||||||
{user_wish}
|
|
||||||
Generate reseach keywords (less than 5 keywords) accordingly.
|
|
||||||
""")
|
|
||||||
gpt_json_io = GptJsonIO(Query)
|
|
||||||
inputs = rf_req + gpt_json_io.format_instructions
|
|
||||||
run_gpt_fn = lambda inputs, sys_prompt: predict_no_ui_long_connection(inputs=inputs, llm_kwargs=llm_kwargs, history=[], sys_prompt=sys_prompt, observe_window=[])
|
|
||||||
analyze_res = run_gpt_fn(inputs, "")
|
|
||||||
logger.info(analyze_res)
|
|
||||||
query: Query = gpt_json_io.generate_output_auto_repair(analyze_res, run_gpt_fn)
|
|
||||||
video_engine_keywords = query.search_keyword
|
|
||||||
# 关键词展示
|
|
||||||
chatbot.append((None, f"检索关键词已确认: {video_engine_keywords}。筛选中, 请稍等..."))
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
|
|
||||||
# 获取候选资源
|
|
||||||
candadate_dictionary: dict = get_video_resource(video_engine_keywords)
|
|
||||||
candadate_dictionary_as_str = json.dumps(candadate_dictionary, ensure_ascii=False, indent=4)
|
|
||||||
|
|
||||||
# 展示候选资源
|
|
||||||
candadate_display = "\n".join([f"{i+1}. {it['title']}" for i, it in enumerate(candadate_dictionary)])
|
|
||||||
chatbot.append((None, f"候选:\n\n{candadate_display}"))
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
|
|
||||||
# 结构化生成
|
|
||||||
rf_req_2 = dedent(f"""
|
|
||||||
The user wish to get the following resource:
|
|
||||||
{user_wish}
|
|
||||||
|
|
||||||
Select the most relevant and suitable video resource from the following search results:
|
|
||||||
{candadate_dictionary_as_str}
|
|
||||||
|
|
||||||
Note:
|
|
||||||
1. The first several search video results are more likely to satisfy the user's wish.
|
|
||||||
2. The time duration of the video should be less than 10 minutes.
|
|
||||||
3. You should analyze the search results first, before giving your answer.
|
|
||||||
4. Use Chinese if possible.
|
|
||||||
5. Beside the primary video selection, give a backup video resource `bvid`.
|
|
||||||
""")
|
|
||||||
gpt_json_io = GptJsonIO(VideoResource)
|
|
||||||
inputs = rf_req_2 + gpt_json_io.format_instructions
|
|
||||||
run_gpt_fn = lambda inputs, sys_prompt: predict_no_ui_long_connection(inputs=inputs, llm_kwargs=llm_kwargs, history=[], sys_prompt=sys_prompt, observe_window=[])
|
|
||||||
analyze_res = run_gpt_fn(inputs, "")
|
|
||||||
logger.info(analyze_res)
|
|
||||||
video_resource: VideoResource = gpt_json_io.generate_output_auto_repair(analyze_res, run_gpt_fn)
|
|
||||||
|
|
||||||
# Display
|
|
||||||
chatbot.append(
|
|
||||||
(None,
|
|
||||||
f"分析：{video_resource.thought}" "<br/>"
|
|
||||||
f"选择: `{video_resource.title}`。" "<br/>"
|
|
||||||
f"作者：{video_resource.author}"
|
|
||||||
)
|
|
||||||
)
|
|
||||||
chatbot.append((None, f"下载中, 请稍等..."))
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
|
|
||||||
if video_resource and video_resource.bvid:
|
|
||||||
logger.info(video_resource)
|
|
||||||
downloaded = yield from download_video(video_resource.bvid, chatbot.get_user(), chatbot, history)
|
|
||||||
if not downloaded:
|
|
||||||
chatbot.append((None, f"下载失败, 尝试备选 ..."))
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
downloaded = yield from download_video(video_resource.another_failsafe_bvid, chatbot.get_user(), chatbot, history)
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@CatchException
|
|
||||||
def debug(bvid, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_request):
|
|
||||||
yield from download_video(bvid, chatbot.get_user(), chatbot, history)
|
|
||||||
@@ -92,7 +92,7 @@ class MiniGame_ResumeStory(GptAcademicGameBaseState):
|
|||||||
|
|
||||||
def generate_story_image(self, story_paragraph):
|
def generate_story_image(self, story_paragraph):
|
||||||
try:
|
try:
|
||||||
from crazy_functions.Image_Generate import gen_image
|
from crazy_functions.AntFin import gen_image
|
||||||
prompt_ = predict_no_ui_long_connection(inputs=story_paragraph, llm_kwargs=self.llm_kwargs, history=[], sys_prompt='你需要根据用户给出的小说段落，进行简短的环境描写。要求：80字以内。')
|
prompt_ = predict_no_ui_long_connection(inputs=story_paragraph, llm_kwargs=self.llm_kwargs, history=[], sys_prompt='你需要根据用户给出的小说段落，进行简短的环境描写。要求：80字以内。')
|
||||||
image_url, image_path = gen_image(self.llm_kwargs, prompt_, '512x512', model="dall-e-2", quality='standard', style='natural')
|
image_url, image_path = gen_image(self.llm_kwargs, prompt_, '512x512', model="dall-e-2", quality='standard', style='natural')
|
||||||
return f'<br/><div align="center"><img src="file={image_path}"></div>'
|
return f'<br/><div align="center"><img src="file={image_path}"></div>'
|
||||||
|
|||||||
@@ -300,8 +300,7 @@ def Latex精细分解与转化(file_manifest, project_folder, llm_kwargs, plugin
|
|||||||
write_html(pfg.sp_file_contents, pfg.sp_file_result, chatbot=chatbot, project_folder=project_folder)
|
write_html(pfg.sp_file_contents, pfg.sp_file_result, chatbot=chatbot, project_folder=project_folder)
|
||||||
|
|
||||||
# <-------- 写出文件 ---------->
|
# <-------- 写出文件 ---------->
|
||||||
model_name = llm_kwargs['llm_model'].replace('_', '\\_') # 替换LLM模型名称中的下划线为转义字符
|
msg = f"当前大语言模型: {llm_kwargs['llm_model']}，当前语言模型温度设定: {llm_kwargs['temperature']}。"
|
||||||
msg = f"当前大语言模型: {model_name}，当前语言模型温度设定: {llm_kwargs['temperature']}。"
|
|
||||||
final_tex = lps.merge_result(pfg.file_result, mode, msg)
|
final_tex = lps.merge_result(pfg.file_result, mode, msg)
|
||||||
objdump((lps, pfg.file_result, mode, msg), file=pj(project_folder,'merge_result.pkl'))
|
objdump((lps, pfg.file_result, mode, msg), file=pj(project_folder,'merge_result.pkl'))
|
||||||
|
|
||||||
@@ -352,41 +351,6 @@ def 编译Latex(chatbot, history, main_file_original, main_file_modified, work_f
|
|||||||
chatbot.append([f"正在编译PDF文档", f'编译已经开始。当前工作路径为{work_folder}，如果程序停顿5分钟以上，请直接去该路径下取回翻译结果，或者重启之后再度尝试 ...']); yield from update_ui(chatbot=chatbot, history=history)
|
chatbot.append([f"正在编译PDF文档", f'编译已经开始。当前工作路径为{work_folder}，如果程序停顿5分钟以上，请直接去该路径下取回翻译结果，或者重启之后再度尝试 ...']); yield from update_ui(chatbot=chatbot, history=history)
|
||||||
chatbot.append([f"正在编译PDF文档", '...']); yield from update_ui(chatbot=chatbot, history=history); time.sleep(1); chatbot[-1] = list(chatbot[-1]) # 刷新界面
|
chatbot.append([f"正在编译PDF文档", '...']); yield from update_ui(chatbot=chatbot, history=history); time.sleep(1); chatbot[-1] = list(chatbot[-1]) # 刷新界面
|
||||||
yield from update_ui_lastest_msg('编译已经开始...', chatbot, history) # 刷新Gradio前端界面
|
yield from update_ui_lastest_msg('编译已经开始...', chatbot, history) # 刷新Gradio前端界面
|
||||||
# 检查是否需要使用xelatex
|
|
||||||
def check_if_need_xelatex(tex_path):
|
|
||||||
try:
|
|
||||||
with open(tex_path, 'r', encoding='utf-8', errors='replace') as f:
|
|
||||||
content = f.read(5000)
|
|
||||||
# 检查是否有使用xelatex的宏包
|
|
||||||
need_xelatex = any(
|
|
||||||
pkg in content
|
|
||||||
for pkg in ['fontspec', 'xeCJK', 'xetex', 'unicode-math', 'xltxtra', 'xunicode']
|
|
||||||
)
|
|
||||||
if need_xelatex:
|
|
||||||
logger.info(f"检测到宏包需要xelatex编译, 切换至xelatex编译")
|
|
||||||
else:
|
|
||||||
logger.info(f"未检测到宏包需要xelatex编译, 使用pdflatex编译")
|
|
||||||
return need_xelatex
|
|
||||||
except Exception:
|
|
||||||
return False
|
|
||||||
|
|
||||||
# 根据编译器类型返回编译命令
|
|
||||||
def get_compile_command(compiler, filename):
|
|
||||||
compile_command = f'{compiler} -interaction=batchmode -file-line-error {filename}.tex'
|
|
||||||
logger.info('Latex 编译指令: ', compile_command)
|
|
||||||
return compile_command
|
|
||||||
|
|
||||||
# 确定使用的编译器
|
|
||||||
compiler = 'pdflatex'
|
|
||||||
if check_if_need_xelatex(pj(work_folder_modified, f'{main_file_modified}.tex')):
|
|
||||||
logger.info("检测到宏包需要xelatex编译，切换至xelatex编译")
|
|
||||||
# Check if xelatex is installed
|
|
||||||
try:
|
|
||||||
import subprocess
|
|
||||||
subprocess.run(['xelatex', '--version'], capture_output=True, check=True)
|
|
||||||
compiler = 'xelatex'
|
|
||||||
except (subprocess.CalledProcessError, FileNotFoundError):
|
|
||||||
raise RuntimeError("检测到需要使用xelatex编译，但系统中未安装xelatex。请先安装texlive或其他提供xelatex的LaTeX发行版。")
|
|
||||||
|
|
||||||
while True:
|
while True:
|
||||||
import os
|
import os
|
||||||
@@ -397,10 +361,10 @@ def 编译Latex(chatbot, history, main_file_original, main_file_modified, work_f
|
|||||||
|
|
||||||
# https://stackoverflow.com/questions/738755/dont-make-me-manually-abort-a-latex-compile-when-theres-an-error
|
# https://stackoverflow.com/questions/738755/dont-make-me-manually-abort-a-latex-compile-when-theres-an-error
|
||||||
yield from update_ui_lastest_msg(f'尝试第 {n_fix}/{max_try} 次编译, 编译原始PDF ...', chatbot, history) # 刷新Gradio前端界面
|
yield from update_ui_lastest_msg(f'尝试第 {n_fix}/{max_try} 次编译, 编译原始PDF ...', chatbot, history) # 刷新Gradio前端界面
|
||||||
ok = compile_latex_with_timeout(get_compile_command(compiler, main_file_original), work_folder_original)
|
ok = compile_latex_with_timeout(f'pdflatex -interaction=batchmode -file-line-error {main_file_original}.tex', work_folder_original)
|
||||||
|
|
||||||
yield from update_ui_lastest_msg(f'尝试第 {n_fix}/{max_try} 次编译, 编译转化后的PDF ...', chatbot, history) # 刷新Gradio前端界面
|
yield from update_ui_lastest_msg(f'尝试第 {n_fix}/{max_try} 次编译, 编译转化后的PDF ...', chatbot, history) # 刷新Gradio前端界面
|
||||||
ok = compile_latex_with_timeout(get_compile_command(compiler, main_file_modified), work_folder_modified)
|
ok = compile_latex_with_timeout(f'pdflatex -interaction=batchmode -file-line-error {main_file_modified}.tex', work_folder_modified)
|
||||||
|
|
||||||
if ok and os.path.exists(pj(work_folder_modified, f'{main_file_modified}.pdf')):
|
if ok and os.path.exists(pj(work_folder_modified, f'{main_file_modified}.pdf')):
|
||||||
# 只有第二步成功，才能继续下面的步骤
|
# 只有第二步成功，才能继续下面的步骤
|
||||||
@@ -411,10 +375,10 @@ def 编译Latex(chatbot, history, main_file_original, main_file_modified, work_f
|
|||||||
ok = compile_latex_with_timeout(f'bibtex {main_file_modified}.aux', work_folder_modified)
|
ok = compile_latex_with_timeout(f'bibtex {main_file_modified}.aux', work_folder_modified)
|
||||||
|
|
||||||
yield from update_ui_lastest_msg(f'尝试第 {n_fix}/{max_try} 次编译, 编译文献交叉引用 ...', chatbot, history) # 刷新Gradio前端界面
|
yield from update_ui_lastest_msg(f'尝试第 {n_fix}/{max_try} 次编译, 编译文献交叉引用 ...', chatbot, history) # 刷新Gradio前端界面
|
||||||
ok = compile_latex_with_timeout(get_compile_command(compiler, main_file_original), work_folder_original)
|
ok = compile_latex_with_timeout(f'pdflatex -interaction=batchmode -file-line-error {main_file_original}.tex', work_folder_original)
|
||||||
ok = compile_latex_with_timeout(get_compile_command(compiler, main_file_modified), work_folder_modified)
|
ok = compile_latex_with_timeout(f'pdflatex -interaction=batchmode -file-line-error {main_file_modified}.tex', work_folder_modified)
|
||||||
ok = compile_latex_with_timeout(get_compile_command(compiler, main_file_original), work_folder_original)
|
ok = compile_latex_with_timeout(f'pdflatex -interaction=batchmode -file-line-error {main_file_original}.tex', work_folder_original)
|
||||||
ok = compile_latex_with_timeout(get_compile_command(compiler, main_file_modified), work_folder_modified)
|
ok = compile_latex_with_timeout(f'pdflatex -interaction=batchmode -file-line-error {main_file_modified}.tex', work_folder_modified)
|
||||||
|
|
||||||
if mode!='translate_zh':
|
if mode!='translate_zh':
|
||||||
yield from update_ui_lastest_msg(f'尝试第 {n_fix}/{max_try} 次编译, 使用latexdiff生成论文转化前后对比 ...', chatbot, history) # 刷新Gradio前端界面
|
yield from update_ui_lastest_msg(f'尝试第 {n_fix}/{max_try} 次编译, 使用latexdiff生成论文转化前后对比 ...', chatbot, history) # 刷新Gradio前端界面
|
||||||
@@ -422,10 +386,10 @@ def 编译Latex(chatbot, history, main_file_original, main_file_modified, work_f
|
|||||||
ok = compile_latex_with_timeout(f'latexdiff --encoding=utf8 --append-safecmd=subfile {work_folder_original}/{main_file_original}.tex {work_folder_modified}/{main_file_modified}.tex --flatten > {work_folder}/merge_diff.tex', os.getcwd())
|
ok = compile_latex_with_timeout(f'latexdiff --encoding=utf8 --append-safecmd=subfile {work_folder_original}/{main_file_original}.tex {work_folder_modified}/{main_file_modified}.tex --flatten > {work_folder}/merge_diff.tex', os.getcwd())
|
||||||
|
|
||||||
yield from update_ui_lastest_msg(f'尝试第 {n_fix}/{max_try} 次编译, 正在编译对比PDF ...', chatbot, history) # 刷新Gradio前端界面
|
yield from update_ui_lastest_msg(f'尝试第 {n_fix}/{max_try} 次编译, 正在编译对比PDF ...', chatbot, history) # 刷新Gradio前端界面
|
||||||
ok = compile_latex_with_timeout(get_compile_command(compiler, 'merge_diff'), work_folder)
|
ok = compile_latex_with_timeout(f'pdflatex -interaction=batchmode -file-line-error merge_diff.tex', work_folder)
|
||||||
ok = compile_latex_with_timeout(f'bibtex merge_diff.aux', work_folder)
|
ok = compile_latex_with_timeout(f'bibtex merge_diff.aux', work_folder)
|
||||||
ok = compile_latex_with_timeout(get_compile_command(compiler, 'merge_diff'), work_folder)
|
ok = compile_latex_with_timeout(f'pdflatex -interaction=batchmode -file-line-error merge_diff.tex', work_folder)
|
||||||
ok = compile_latex_with_timeout(get_compile_command(compiler, 'merge_diff'), work_folder)
|
ok = compile_latex_with_timeout(f'pdflatex -interaction=batchmode -file-line-error merge_diff.tex', work_folder)
|
||||||
|
|
||||||
# <---------- 检查结果 ----------->
|
# <---------- 检查结果 ----------->
|
||||||
results_ = ""
|
results_ = ""
|
||||||
|
|||||||
@@ -1,15 +1,13 @@
|
|||||||
from toolbox import update_ui, get_conf, promote_file_to_downloadzone, update_ui_lastest_msg, generate_file_link
|
from toolbox import update_ui, get_conf, promote_file_to_downloadzone, update_ui_lastest_msg, generate_file_link
|
||||||
from shared_utils.docker_as_service_api import stream_daas
|
from shared_utils.docker_as_service_api import stream_daas
|
||||||
from shared_utils.docker_as_service_api import DockerServiceApiComModel
|
from shared_utils.docker_as_service_api import DockerServiceApiComModel
|
||||||
import random
|
|
||||||
|
|
||||||
def download_video(video_id, only_audio, user_name, chatbot, history):
|
def download_video(video_id, only_audio, user_name, chatbot, history):
|
||||||
from toolbox import get_log_folder
|
from toolbox import get_log_folder
|
||||||
chatbot.append([None, "Processing..."])
|
chatbot.append([None, "Processing..."])
|
||||||
yield from update_ui(chatbot, history)
|
yield from update_ui(chatbot, history)
|
||||||
client_command = f'{video_id} --audio-only' if only_audio else video_id
|
client_command = f'{video_id} --audio-only' if only_audio else video_id
|
||||||
server_urls = get_conf('DAAS_SERVER_URLS')
|
server_url = get_conf('DAAS_SERVER_URL')
|
||||||
server_url = random.choice(server_urls)
|
|
||||||
docker_service_api_com_model = DockerServiceApiComModel(client_command=client_command)
|
docker_service_api_com_model = DockerServiceApiComModel(client_command=client_command)
|
||||||
save_file_dir = get_log_folder(user_name, plugin_name='media_downloader')
|
save_file_dir = get_log_folder(user_name, plugin_name='media_downloader')
|
||||||
for output_manifest in stream_daas(docker_service_api_com_model, server_url, save_file_dir):
|
for output_manifest in stream_daas(docker_service_api_com_model, server_url, save_file_dir):
|
||||||
@@ -33,9 +31,7 @@ def download_video(video_id, only_audio, user_name, chatbot, history):
|
|||||||
def search_videos(keywords):
|
def search_videos(keywords):
|
||||||
from toolbox import get_log_folder
|
from toolbox import get_log_folder
|
||||||
client_command = keywords
|
client_command = keywords
|
||||||
server_urls = get_conf('DAAS_SERVER_URLS')
|
server_url = get_conf('DAAS_SERVER_URL').replace('stream', 'search')
|
||||||
server_url = random.choice(server_urls)
|
|
||||||
server_url = server_url.replace('stream', 'search')
|
|
||||||
docker_service_api_com_model = DockerServiceApiComModel(client_command=client_command)
|
docker_service_api_com_model = DockerServiceApiComModel(client_command=client_command)
|
||||||
save_file_dir = get_log_folder("default_user", plugin_name='media_downloader')
|
save_file_dir = get_log_folder("default_user", plugin_name='media_downloader')
|
||||||
for output_manifest in stream_daas(docker_service_api_com_model, server_url, save_file_dir):
|
for output_manifest in stream_daas(docker_service_api_com_model, server_url, save_file_dir):
|
||||||
|
|||||||
@@ -6,128 +6,75 @@ from crazy_functions.crazy_utils import get_files_from_everything
|
|||||||
from shared_utils.colorful import *
|
from shared_utils.colorful import *
|
||||||
from loguru import logger
|
from loguru import logger
|
||||||
import os
|
import os
|
||||||
import requests
|
|
||||||
import time
|
import time
|
||||||
|
|
||||||
|
def refresh_key(doc2x_api_key):
|
||||||
|
import requests, json
|
||||||
|
url = "https://api.doc2x.noedgeai.com/api/token/refresh"
|
||||||
|
res = requests.post(
|
||||||
|
url,
|
||||||
|
headers={"Authorization": "Bearer " + doc2x_api_key}
|
||||||
|
)
|
||||||
|
res_json = []
|
||||||
|
if res.status_code == 200:
|
||||||
|
decoded = res.content.decode("utf-8")
|
||||||
|
res_json = json.loads(decoded)
|
||||||
|
doc2x_api_key = res_json['data']['token']
|
||||||
|
else:
|
||||||
|
raise RuntimeError(format("[ERROR] status code: %d, body: %s" % (res.status_code, res.text)))
|
||||||
|
return doc2x_api_key
|
||||||
|
|
||||||
def retry_request(max_retries=3, delay=3):
|
|
||||||
"""
|
|
||||||
Decorator for retrying HTTP requests
|
|
||||||
Args:
|
|
||||||
max_retries: Maximum number of retry attempts
|
|
||||||
delay: Delay between retries in seconds
|
|
||||||
"""
|
|
||||||
|
|
||||||
def decorator(func):
|
|
||||||
def wrapper(*args, **kwargs):
|
|
||||||
for attempt in range(max_retries):
|
|
||||||
try:
|
|
||||||
return func(*args, **kwargs)
|
|
||||||
except Exception as e:
|
|
||||||
if attempt < max_retries - 1:
|
|
||||||
logger.error(
|
|
||||||
f"Request failed, retrying... ({attempt + 1}/{max_retries}) Error: {e}"
|
|
||||||
)
|
|
||||||
time.sleep(delay)
|
|
||||||
continue
|
|
||||||
raise e
|
|
||||||
return None
|
|
||||||
|
|
||||||
return wrapper
|
|
||||||
|
|
||||||
return decorator
|
|
||||||
|
|
||||||
|
|
||||||
@retry_request()
|
|
||||||
def make_request(method, url, **kwargs):
|
|
||||||
"""
|
|
||||||
Make HTTP request with retry mechanism
|
|
||||||
"""
|
|
||||||
return requests.request(method, url, **kwargs)
|
|
||||||
|
|
||||||
|
|
||||||
def doc2x_api_response_status(response, uid=""):
|
|
||||||
"""
|
|
||||||
Check the status of Doc2x API response
|
|
||||||
Args:
|
|
||||||
response_data: Response object from Doc2x API
|
|
||||||
"""
|
|
||||||
response_json = response.json()
|
|
||||||
response_data = response_json.get("data", {})
|
|
||||||
code = response_json.get("code", "Unknown")
|
|
||||||
meg = response_data.get("message", response_json)
|
|
||||||
trace_id = response.headers.get("trace-id", "Failed to get trace-id")
|
|
||||||
if response.status_code != 200:
|
|
||||||
raise RuntimeError(
|
|
||||||
f"Doc2x return an error:\nTrace ID: {trace_id} {uid}\n{response.status_code} - {response_json}"
|
|
||||||
)
|
|
||||||
if code in ["parse_page_limit_exceeded", "parse_concurrency_limit"]:
|
|
||||||
raise RuntimeError(
|
|
||||||
f"Reached the limit of Doc2x:\nTrace ID: {trace_id} {uid}\n{code} - {meg}"
|
|
||||||
)
|
|
||||||
if code not in ["ok", "success"]:
|
|
||||||
raise RuntimeError(
|
|
||||||
f"Doc2x return an error:\nTrace ID: {trace_id} {uid}\n{code} - {meg}"
|
|
||||||
)
|
|
||||||
return response_data
|
|
||||||
|
|
||||||
|
|
||||||
def 解析PDF_DOC2X_转Latex(pdf_file_path):
|
def 解析PDF_DOC2X_转Latex(pdf_file_path):
|
||||||
zip_file_path, unzipped_folder = 解析PDF_DOC2X(pdf_file_path, format="tex")
|
zip_file_path, unzipped_folder = 解析PDF_DOC2X(pdf_file_path, format='tex')
|
||||||
return unzipped_folder
|
return unzipped_folder
|
||||||
|
|
||||||
|
|
||||||
def 解析PDF_DOC2X(pdf_file_path, format="tex"):
|
def 解析PDF_DOC2X(pdf_file_path, format='tex'):
|
||||||
"""
|
"""
|
||||||
format: 'tex', 'md', 'docx'
|
format: 'tex', 'md', 'docx'
|
||||||
"""
|
"""
|
||||||
|
import requests, json, os
|
||||||
DOC2X_API_KEY = get_conf("DOC2X_API_KEY")
|
DOC2X_API_KEY = get_conf('DOC2X_API_KEY')
|
||||||
latex_dir = get_log_folder(plugin_name="pdf_ocr_latex")
|
latex_dir = get_log_folder(plugin_name="pdf_ocr_latex")
|
||||||
markdown_dir = get_log_folder(plugin_name="pdf_ocr")
|
markdown_dir = get_log_folder(plugin_name="pdf_ocr")
|
||||||
doc2x_api_key = DOC2X_API_KEY
|
doc2x_api_key = DOC2X_API_KEY
|
||||||
|
|
||||||
# < ------ 第1步：预上传获取URL，然后上传文件 ------ >
|
|
||||||
logger.info("Doc2x 上传文件：预上传获取URL")
|
|
||||||
res = make_request(
|
|
||||||
"POST",
|
|
||||||
"https://v2.doc2x.noedgeai.com/api/v2/parse/preupload",
|
|
||||||
headers={"Authorization": "Bearer " + doc2x_api_key},
|
|
||||||
timeout=15,
|
|
||||||
)
|
|
||||||
res_data = doc2x_api_response_status(res)
|
|
||||||
upload_url = res_data["url"]
|
|
||||||
uuid = res_data["uid"]
|
|
||||||
|
|
||||||
logger.info("Doc2x 上传文件：上传文件")
|
# < ------ 第1步：上传 ------ >
|
||||||
with open(pdf_file_path, "rb") as file:
|
logger.info("Doc2x 第1步：上传")
|
||||||
res = make_request("PUT", upload_url, data=file, timeout=60)
|
with open(pdf_file_path, 'rb') as file:
|
||||||
res.raise_for_status()
|
res = requests.post(
|
||||||
|
"https://v2.doc2x.noedgeai.com/api/v2/parse/pdf",
|
||||||
|
headers={"Authorization": "Bearer " + doc2x_api_key},
|
||||||
|
data=file
|
||||||
|
)
|
||||||
|
# res_json = []
|
||||||
|
if res.status_code == 200:
|
||||||
|
res_json = res.json()
|
||||||
|
else:
|
||||||
|
raise RuntimeError(f"Doc2x return an error: {res.json()}")
|
||||||
|
uuid = res_json['data']['uid']
|
||||||
|
|
||||||
# < ------ 第2步：轮询等待 ------ >
|
# < ------ 第2步：轮询等待 ------ >
|
||||||
logger.info("Doc2x 处理文件中：轮询等待")
|
logger.info("Doc2x 第2步：轮询等待")
|
||||||
params = {"uid": uuid}
|
params = {'uid': uuid}
|
||||||
max_attempts = 60
|
while True:
|
||||||
attempt = 0
|
res = requests.get(
|
||||||
while attempt < max_attempts:
|
'https://v2.doc2x.noedgeai.com/api/v2/parse/status',
|
||||||
res = make_request(
|
|
||||||
"GET",
|
|
||||||
"https://v2.doc2x.noedgeai.com/api/v2/parse/status",
|
|
||||||
headers={"Authorization": "Bearer " + doc2x_api_key},
|
headers={"Authorization": "Bearer " + doc2x_api_key},
|
||||||
params=params,
|
params=params
|
||||||
timeout=15,
|
|
||||||
)
|
)
|
||||||
res_data = doc2x_api_response_status(res)
|
res_json = res.json()
|
||||||
if res_data["status"] == "success":
|
if res_json['data']['status'] == "success":
|
||||||
break
|
break
|
||||||
elif res_data["status"] == "processing":
|
elif res_json['data']['status'] == "processing":
|
||||||
time.sleep(5)
|
time.sleep(3)
|
||||||
logger.info(f"Doc2x is processing at {res_data['progress']}%")
|
logger.info(f"Doc2x is processing at {res_json['data']['progress']}%")
|
||||||
attempt += 1
|
elif res_json['data']['status'] == "failed":
|
||||||
else:
|
raise RuntimeError(f"Doc2x return an error: {res_json}")
|
||||||
raise RuntimeError(f"Doc2x return an error: {res_data}")
|
|
||||||
if attempt >= max_attempts:
|
|
||||||
raise RuntimeError("Doc2x processing timeout after maximum attempts")
|
|
||||||
|
|
||||||
# < ------ 第3步：提交转化 ------ >
|
# < ------ 第3步：提交转化 ------ >
|
||||||
logger.info("Doc2x 第3步：提交转化")
|
logger.info("Doc2x 第3步：提交转化")
|
||||||
@@ -137,44 +84,42 @@ def 解析PDF_DOC2X(pdf_file_path, format="tex"):
|
|||||||
"formula_mode": "dollar",
|
"formula_mode": "dollar",
|
||||||
"filename": "output"
|
"filename": "output"
|
||||||
}
|
}
|
||||||
res = make_request(
|
res = requests.post(
|
||||||
"POST",
|
'https://v2.doc2x.noedgeai.com/api/v2/convert/parse',
|
||||||
"https://v2.doc2x.noedgeai.com/api/v2/convert/parse",
|
|
||||||
headers={"Authorization": "Bearer " + doc2x_api_key},
|
headers={"Authorization": "Bearer " + doc2x_api_key},
|
||||||
json=data,
|
json=data
|
||||||
timeout=15,
|
|
||||||
)
|
)
|
||||||
doc2x_api_response_status(res, uid=f"uid: {uuid}")
|
if res.status_code == 200:
|
||||||
|
res_json = res.json()
|
||||||
|
else:
|
||||||
|
raise RuntimeError(f"Doc2x return an error: {res.json()}")
|
||||||
|
|
||||||
|
|
||||||
# < ------ 第4步：等待结果 ------ >
|
# < ------ 第4步：等待结果 ------ >
|
||||||
logger.info("Doc2x 第4步：等待结果")
|
logger.info("Doc2x 第4步：等待结果")
|
||||||
params = {"uid": uuid}
|
params = {'uid': uuid}
|
||||||
max_attempts = 36
|
while True:
|
||||||
attempt = 0
|
res = requests.get(
|
||||||
while attempt < max_attempts:
|
'https://v2.doc2x.noedgeai.com/api/v2/convert/parse/result',
|
||||||
res = make_request(
|
|
||||||
"GET",
|
|
||||||
"https://v2.doc2x.noedgeai.com/api/v2/convert/parse/result",
|
|
||||||
headers={"Authorization": "Bearer " + doc2x_api_key},
|
headers={"Authorization": "Bearer " + doc2x_api_key},
|
||||||
params=params,
|
params=params
|
||||||
timeout=15,
|
|
||||||
)
|
)
|
||||||
res_data = doc2x_api_response_status(res, uid=f"uid: {uuid}")
|
res_json = res.json()
|
||||||
if res_data["status"] == "success":
|
if res_json['data']['status'] == "success":
|
||||||
break
|
break
|
||||||
elif res_data["status"] == "processing":
|
elif res_json['data']['status'] == "processing":
|
||||||
time.sleep(3)
|
time.sleep(3)
|
||||||
logger.info("Doc2x still processing to convert file")
|
logger.info(f"Doc2x still processing")
|
||||||
attempt += 1
|
elif res_json['data']['status'] == "failed":
|
||||||
if attempt >= max_attempts:
|
raise RuntimeError(f"Doc2x return an error: {res_json}")
|
||||||
raise RuntimeError("Doc2x conversion timeout after maximum attempts")
|
|
||||||
|
|
||||||
# < ------ 第5步：最后的处理 ------ >
|
# < ------ 第5步：最后的处理 ------ >
|
||||||
logger.info("Doc2x 第5步：下载转换后的文件")
|
logger.info("Doc2x 第5步：最后的处理")
|
||||||
|
|
||||||
if format == "tex":
|
if format=='tex':
|
||||||
target_path = latex_dir
|
target_path = latex_dir
|
||||||
if format == "md":
|
if format=='md':
|
||||||
target_path = markdown_dir
|
target_path = markdown_dir
|
||||||
os.makedirs(target_path, exist_ok=True)
|
os.makedirs(target_path, exist_ok=True)
|
||||||
|
|
||||||
@@ -182,18 +127,17 @@ def 解析PDF_DOC2X(pdf_file_path, format="tex"):
|
|||||||
# < ------ 下载 ------ >
|
# < ------ 下载 ------ >
|
||||||
for attempt in range(max_attempt):
|
for attempt in range(max_attempt):
|
||||||
try:
|
try:
|
||||||
result_url = res_data["url"]
|
result_url = res_json['data']['url']
|
||||||
res = make_request("GET", result_url, timeout=60)
|
res = requests.get(result_url)
|
||||||
zip_path = os.path.join(target_path, gen_time_str() + ".zip")
|
zip_path = os.path.join(target_path, gen_time_str() + '.zip')
|
||||||
unzip_path = os.path.join(target_path, gen_time_str())
|
unzip_path = os.path.join(target_path, gen_time_str())
|
||||||
if res.status_code == 200:
|
if res.status_code == 200:
|
||||||
with open(zip_path, "wb") as f:
|
with open(zip_path, "wb") as f: f.write(res.content)
|
||||||
f.write(res.content)
|
|
||||||
else:
|
else:
|
||||||
raise RuntimeError(f"Doc2x return an error: {res.json()}")
|
raise RuntimeError(f"Doc2x return an error: {res.json()}")
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
if attempt < max_attempt - 1:
|
if attempt < max_attempt - 1:
|
||||||
logger.error(f"Failed to download uid = {uuid} file, retrying... {e}")
|
logger.error(f"Failed to download latex file, retrying... {e}")
|
||||||
time.sleep(3)
|
time.sleep(3)
|
||||||
continue
|
continue
|
||||||
else:
|
else:
|
||||||
@@ -201,31 +145,22 @@ def 解析PDF_DOC2X(pdf_file_path, format="tex"):
|
|||||||
|
|
||||||
# < ------ 解压 ------ >
|
# < ------ 解压 ------ >
|
||||||
import zipfile
|
import zipfile
|
||||||
with zipfile.ZipFile(zip_path, "r") as zip_ref:
|
with zipfile.ZipFile(zip_path, 'r') as zip_ref:
|
||||||
zip_ref.extractall(unzip_path)
|
zip_ref.extractall(unzip_path)
|
||||||
return zip_path, unzip_path
|
return zip_path, unzip_path
|
||||||
|
|
||||||
|
|
||||||
def 解析PDF_DOC2X_单文件(
|
def 解析PDF_DOC2X_单文件(fp, project_folder, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, DOC2X_API_KEY, user_request):
|
||||||
fp,
|
|
||||||
project_folder,
|
|
||||||
llm_kwargs,
|
|
||||||
plugin_kwargs,
|
|
||||||
chatbot,
|
|
||||||
history,
|
|
||||||
system_prompt,
|
|
||||||
DOC2X_API_KEY,
|
|
||||||
user_request,
|
|
||||||
):
|
|
||||||
def pdf2markdown(filepath):
|
def pdf2markdown(filepath):
|
||||||
chatbot.append((None, f"Doc2x 解析中"))
|
chatbot.append((None, f"Doc2x 解析中"))
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
||||||
|
|
||||||
md_zip_path, unzipped_folder = 解析PDF_DOC2X(filepath, format="md")
|
md_zip_path, unzipped_folder = 解析PDF_DOC2X(filepath, format='md')
|
||||||
|
|
||||||
promote_file_to_downloadzone(md_zip_path, chatbot=chatbot)
|
promote_file_to_downloadzone(md_zip_path, chatbot=chatbot)
|
||||||
chatbot.append((None, f"完成解析 {md_zip_path} ..."))
|
chatbot.append((None, f"完成解析 {md_zip_path} ..."))
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
||||||
return md_zip_path
|
return md_zip_path
|
||||||
|
|
||||||
def deliver_to_markdown_plugin(md_zip_path, user_request):
|
def deliver_to_markdown_plugin(md_zip_path, user_request):
|
||||||
@@ -239,97 +174,77 @@ def 解析PDF_DOC2X_单文件(
|
|||||||
os.makedirs(target_path_base, exist_ok=True)
|
os.makedirs(target_path_base, exist_ok=True)
|
||||||
shutil.copyfile(md_zip_path, this_file_path)
|
shutil.copyfile(md_zip_path, this_file_path)
|
||||||
ex_folder = this_file_path + ".extract"
|
ex_folder = this_file_path + ".extract"
|
||||||
extract_archive(file_path=this_file_path, dest_dir=ex_folder)
|
extract_archive(
|
||||||
|
file_path=this_file_path, dest_dir=ex_folder
|
||||||
|
)
|
||||||
|
|
||||||
# edit markdown files
|
# edit markdown files
|
||||||
success, file_manifest, project_folder = get_files_from_everything(
|
success, file_manifest, project_folder = get_files_from_everything(ex_folder, type='.md')
|
||||||
ex_folder, type=".md"
|
|
||||||
)
|
|
||||||
for generated_fp in file_manifest:
|
for generated_fp in file_manifest:
|
||||||
# 修正一些公式问题
|
# 修正一些公式问题
|
||||||
with open(generated_fp, "r", encoding="utf8") as f:
|
with open(generated_fp, 'r', encoding='utf8') as f:
|
||||||
content = f.read()
|
content = f.read()
|
||||||
# 将公式中的\[ \]替换成$$
|
# 将公式中的\[ \]替换成$$
|
||||||
content = content.replace(r"\[", r"$$").replace(r"\]", r"$$")
|
content = content.replace(r'\[', r'$$').replace(r'\]', r'$$')
|
||||||
# 将公式中的\( \)替换成$
|
# 将公式中的\( \)替换成$
|
||||||
content = content.replace(r"\(", r"$").replace(r"\)", r"$")
|
content = content.replace(r'\(', r'$').replace(r'\)', r'$')
|
||||||
content = content.replace("```markdown", "\n").replace("```", "\n")
|
content = content.replace('```markdown', '\n').replace('```', '\n')
|
||||||
with open(generated_fp, "w", encoding="utf8") as f:
|
with open(generated_fp, 'w', encoding='utf8') as f:
|
||||||
f.write(content)
|
f.write(content)
|
||||||
promote_file_to_downloadzone(generated_fp, chatbot=chatbot)
|
promote_file_to_downloadzone(generated_fp, chatbot=chatbot)
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
||||||
|
|
||||||
# 生成在线预览html
|
# 生成在线预览html
|
||||||
file_name = "在线预览翻译（原文）" + gen_time_str() + ".html"
|
file_name = '在线预览翻译（原文）' + gen_time_str() + '.html'
|
||||||
preview_fp = os.path.join(ex_folder, file_name)
|
preview_fp = os.path.join(ex_folder, file_name)
|
||||||
from shared_utils.advanced_markdown_format import (
|
from shared_utils.advanced_markdown_format import markdown_convertion_for_file
|
||||||
markdown_convertion_for_file,
|
|
||||||
)
|
|
||||||
|
|
||||||
with open(generated_fp, "r", encoding="utf-8") as f:
|
with open(generated_fp, "r", encoding="utf-8") as f:
|
||||||
md = f.read()
|
md = f.read()
|
||||||
# # Markdown中使用不标准的表格，需要在表格前加上一个emoji，以便公式渲染
|
# # Markdown中使用不标准的表格，需要在表格前加上一个emoji，以便公式渲染
|
||||||
# md = re.sub(r'^<table>', r'.<table>', md, flags=re.MULTILINE)
|
# md = re.sub(r'^<table>', r'.<table>', md, flags=re.MULTILINE)
|
||||||
html = markdown_convertion_for_file(md)
|
html = markdown_convertion_for_file(md)
|
||||||
with open(preview_fp, "w", encoding="utf-8") as f:
|
with open(preview_fp, "w", encoding="utf-8") as f: f.write(html)
|
||||||
f.write(html)
|
|
||||||
chatbot.append([None, f"生成在线预览：{generate_file_link([preview_fp])}"])
|
chatbot.append([None, f"生成在线预览：{generate_file_link([preview_fp])}"])
|
||||||
promote_file_to_downloadzone(preview_fp, chatbot=chatbot)
|
promote_file_to_downloadzone(preview_fp, chatbot=chatbot)
|
||||||
|
|
||||||
chatbot.append((None, f"调用Markdown插件 {ex_folder} ..."))
|
|
||||||
plugin_kwargs["markdown_expected_output_dir"] = ex_folder
|
|
||||||
|
|
||||||
translated_f_name = "translated_markdown.md"
|
|
||||||
generated_fp = plugin_kwargs["markdown_expected_output_path"] = os.path.join(
|
chatbot.append((None, f"调用Markdown插件 {ex_folder} ..."))
|
||||||
ex_folder, translated_f_name
|
plugin_kwargs['markdown_expected_output_dir'] = ex_folder
|
||||||
)
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
translated_f_name = 'translated_markdown.md'
|
||||||
yield from Markdown英译中(
|
generated_fp = plugin_kwargs['markdown_expected_output_path'] = os.path.join(ex_folder, translated_f_name)
|
||||||
ex_folder,
|
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
||||||
llm_kwargs,
|
yield from Markdown英译中(ex_folder, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_request)
|
||||||
plugin_kwargs,
|
|
||||||
chatbot,
|
|
||||||
history,
|
|
||||||
system_prompt,
|
|
||||||
user_request,
|
|
||||||
)
|
|
||||||
if os.path.exists(generated_fp):
|
if os.path.exists(generated_fp):
|
||||||
# 修正一些公式问题
|
# 修正一些公式问题
|
||||||
with open(generated_fp, "r", encoding="utf8") as f:
|
with open(generated_fp, 'r', encoding='utf8') as f: content = f.read()
|
||||||
content = f.read()
|
content = content.replace('```markdown', '\n').replace('```', '\n')
|
||||||
content = content.replace("```markdown", "\n").replace("```", "\n")
|
|
||||||
# Markdown中使用不标准的表格，需要在表格前加上一个emoji，以便公式渲染
|
# Markdown中使用不标准的表格，需要在表格前加上一个emoji，以便公式渲染
|
||||||
# content = re.sub(r'^<table>', r'.<table>', content, flags=re.MULTILINE)
|
# content = re.sub(r'^<table>', r'.<table>', content, flags=re.MULTILINE)
|
||||||
with open(generated_fp, "w", encoding="utf8") as f:
|
with open(generated_fp, 'w', encoding='utf8') as f: f.write(content)
|
||||||
f.write(content)
|
|
||||||
# 生成在线预览html
|
# 生成在线预览html
|
||||||
file_name = "在线预览翻译" + gen_time_str() + ".html"
|
file_name = '在线预览翻译' + gen_time_str() + '.html'
|
||||||
preview_fp = os.path.join(ex_folder, file_name)
|
preview_fp = os.path.join(ex_folder, file_name)
|
||||||
from shared_utils.advanced_markdown_format import (
|
from shared_utils.advanced_markdown_format import markdown_convertion_for_file
|
||||||
markdown_convertion_for_file,
|
|
||||||
)
|
|
||||||
|
|
||||||
with open(generated_fp, "r", encoding="utf-8") as f:
|
with open(generated_fp, "r", encoding="utf-8") as f:
|
||||||
md = f.read()
|
md = f.read()
|
||||||
html = markdown_convertion_for_file(md)
|
html = markdown_convertion_for_file(md)
|
||||||
with open(preview_fp, "w", encoding="utf-8") as f:
|
with open(preview_fp, "w", encoding="utf-8") as f: f.write(html)
|
||||||
f.write(html)
|
|
||||||
promote_file_to_downloadzone(preview_fp, chatbot=chatbot)
|
promote_file_to_downloadzone(preview_fp, chatbot=chatbot)
|
||||||
# 生成包含图片的压缩包
|
# 生成包含图片的压缩包
|
||||||
dest_folder = get_log_folder(chatbot.get_user())
|
dest_folder = get_log_folder(chatbot.get_user())
|
||||||
zip_name = "翻译后的带图文档.zip"
|
zip_name = '翻译后的带图文档.zip'
|
||||||
zip_folder(
|
zip_folder(source_folder=ex_folder, dest_folder=dest_folder, zip_name=zip_name)
|
||||||
source_folder=ex_folder, dest_folder=dest_folder, zip_name=zip_name
|
|
||||||
)
|
|
||||||
zip_fp = os.path.join(dest_folder, zip_name)
|
zip_fp = os.path.join(dest_folder, zip_name)
|
||||||
promote_file_to_downloadzone(zip_fp, chatbot=chatbot)
|
promote_file_to_downloadzone(zip_fp, chatbot=chatbot)
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
||||||
|
|
||||||
md_zip_path = yield from pdf2markdown(fp)
|
md_zip_path = yield from pdf2markdown(fp)
|
||||||
yield from deliver_to_markdown_plugin(md_zip_path, user_request)
|
yield from deliver_to_markdown_plugin(md_zip_path, user_request)
|
||||||
|
|
||||||
|
|
||||||
def 解析PDF_基于DOC2X(file_manifest, *args):
|
def 解析PDF_基于DOC2X(file_manifest, *args):
|
||||||
for index, fp in enumerate(file_manifest):
|
for index, fp in enumerate(file_manifest):
|
||||||
yield from 解析PDF_DOC2X_单文件(fp, *args)
|
yield from 解析PDF_DOC2X_单文件(fp, *args)
|
||||||
return
|
return
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -1,180 +0,0 @@
|
|||||||
import re, requests, unicodedata, os
|
|
||||||
from toolbox import update_ui, get_log_folder
|
|
||||||
from toolbox import write_history_to_file, promote_file_to_downloadzone
|
|
||||||
from toolbox import CatchException, report_exception, get_conf
|
|
||||||
from crazy_functions.crazy_utils import request_gpt_model_in_new_thread_with_ui_alive
|
|
||||||
from loguru import logger
|
|
||||||
|
|
||||||
def download_arxiv_(url_pdf):
|
|
||||||
if 'arxiv.org' not in url_pdf:
|
|
||||||
if ('.' in url_pdf) and ('/' not in url_pdf):
|
|
||||||
new_url = 'https://arxiv.org/abs/'+url_pdf
|
|
||||||
logger.info('下载编号：', url_pdf, '自动定位：', new_url)
|
|
||||||
# download_arxiv_(new_url)
|
|
||||||
return download_arxiv_(new_url)
|
|
||||||
else:
|
|
||||||
logger.info('不能识别的URL！')
|
|
||||||
return None
|
|
||||||
if 'abs' in url_pdf:
|
|
||||||
url_pdf = url_pdf.replace('abs', 'pdf')
|
|
||||||
url_pdf = url_pdf + '.pdf'
|
|
||||||
|
|
||||||
url_abs = url_pdf.replace('.pdf', '').replace('pdf', 'abs')
|
|
||||||
title, other_info = get_name(_url_=url_abs)
|
|
||||||
|
|
||||||
paper_id = title.split()[0] # '[1712.00559]'
|
|
||||||
if '2' in other_info['year']:
|
|
||||||
title = other_info['year'] + ' ' + title
|
|
||||||
|
|
||||||
known_conf = ['NeurIPS', 'NIPS', 'Nature', 'Science', 'ICLR', 'AAAI']
|
|
||||||
for k in known_conf:
|
|
||||||
if k in other_info['comment']:
|
|
||||||
title = k + ' ' + title
|
|
||||||
|
|
||||||
download_dir = get_log_folder(plugin_name='arxiv')
|
|
||||||
os.makedirs(download_dir, exist_ok=True)
|
|
||||||
|
|
||||||
title_str = title.replace('?', '？')\
|
|
||||||
.replace(':', '：')\
|
|
||||||
.replace('\"', '“')\
|
|
||||||
.replace('\n', '')\
|
|
||||||
.replace(' ', ' ')\
|
|
||||||
.replace(' ', ' ')
|
|
||||||
|
|
||||||
requests_pdf_url = url_pdf
|
|
||||||
file_path = download_dir+title_str
|
|
||||||
|
|
||||||
logger.info('下载中')
|
|
||||||
proxies = get_conf('proxies')
|
|
||||||
r = requests.get(requests_pdf_url, proxies=proxies)
|
|
||||||
with open(file_path, 'wb+') as f:
|
|
||||||
f.write(r.content)
|
|
||||||
logger.info('下载完成')
|
|
||||||
|
|
||||||
x = "%s %s %s.bib" % (paper_id, other_info['year'], other_info['authors'])
|
|
||||||
x = x.replace('?', '？')\
|
|
||||||
.replace(':', '：')\
|
|
||||||
.replace('\"', '“')\
|
|
||||||
.replace('\n', '')\
|
|
||||||
.replace(' ', ' ')\
|
|
||||||
.replace(' ', ' ')
|
|
||||||
return file_path, other_info
|
|
||||||
|
|
||||||
|
|
||||||
def get_name(_url_):
|
|
||||||
from bs4 import BeautifulSoup
|
|
||||||
logger.info('正在获取文献名！')
|
|
||||||
logger.info(_url_)
|
|
||||||
|
|
||||||
proxies = get_conf('proxies')
|
|
||||||
res = requests.get(_url_, proxies=proxies)
|
|
||||||
|
|
||||||
bs = BeautifulSoup(res.text, 'html.parser')
|
|
||||||
other_details = {}
|
|
||||||
|
|
||||||
# get year
|
|
||||||
try:
|
|
||||||
year = bs.find_all(class_='dateline')[0].text
|
|
||||||
year = re.search(r'(\d{4})', year, re.M | re.I).group(1)
|
|
||||||
other_details['year'] = year
|
|
||||||
abstract = bs.find_all(class_='abstract mathjax')[0].text
|
|
||||||
other_details['abstract'] = abstract
|
|
||||||
except:
|
|
||||||
other_details['year'] = ''
|
|
||||||
logger.info('年份获取失败')
|
|
||||||
|
|
||||||
# get author
|
|
||||||
try:
|
|
||||||
authors = bs.find_all(class_='authors')[0].text
|
|
||||||
authors = authors.split('Authors:')[1]
|
|
||||||
other_details['authors'] = authors
|
|
||||||
except:
|
|
||||||
other_details['authors'] = ''
|
|
||||||
logger.info('authors获取失败')
|
|
||||||
|
|
||||||
# get comment
|
|
||||||
try:
|
|
||||||
comment = bs.find_all(class_='metatable')[0].text
|
|
||||||
real_comment = None
|
|
||||||
for item in comment.replace('\n', ' ').split(' '):
|
|
||||||
if 'Comments' in item:
|
|
||||||
real_comment = item
|
|
||||||
if real_comment is not None:
|
|
||||||
other_details['comment'] = real_comment
|
|
||||||
else:
|
|
||||||
other_details['comment'] = ''
|
|
||||||
except:
|
|
||||||
other_details['comment'] = ''
|
|
||||||
logger.info('年份获取失败')
|
|
||||||
|
|
||||||
title_str = BeautifulSoup(
|
|
||||||
res.text, 'html.parser').find('title').contents[0]
|
|
||||||
logger.info('获取成功：', title_str)
|
|
||||||
# arxiv_recall[_url_] = (title_str+'.pdf', other_details)
|
|
||||||
# with open('./arxiv_recall.pkl', 'wb') as f:
|
|
||||||
# pickle.dump(arxiv_recall, f)
|
|
||||||
|
|
||||||
return title_str+'.pdf', other_details
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@CatchException
|
|
||||||
def 下载arxiv论文并翻译摘要(txt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_request):
|
|
||||||
|
|
||||||
CRAZY_FUNCTION_INFO = "下载arxiv论文并翻译摘要，函数插件作者[binary-husky]。正在提取摘要并下载PDF文档……"
|
|
||||||
import glob
|
|
||||||
import os
|
|
||||||
|
|
||||||
# 基本信息：功能、贡献者
|
|
||||||
chatbot.append(["函数插件功能？", CRAZY_FUNCTION_INFO])
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
|
|
||||||
# 尝试导入依赖，如果缺少依赖，则给出安装建议
|
|
||||||
try:
|
|
||||||
import bs4
|
|
||||||
except:
|
|
||||||
report_exception(chatbot, history,
|
|
||||||
a = f"解析项目: {txt}",
|
|
||||||
b = f"导入软件依赖失败。使用该模块需要额外依赖，安装方法```pip install --upgrade beautifulsoup4```。")
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
|
|
||||||
# 清空历史，以免输入溢出
|
|
||||||
history = []
|
|
||||||
|
|
||||||
# 提取摘要，下载PDF文档
|
|
||||||
try:
|
|
||||||
pdf_path, info = download_arxiv_(txt)
|
|
||||||
except:
|
|
||||||
report_exception(chatbot, history,
|
|
||||||
a = f"解析项目: {txt}",
|
|
||||||
b = f"下载pdf文件未成功")
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
|
|
||||||
# 翻译摘要等
|
|
||||||
i_say = f"请你阅读以下学术论文相关的材料，提取摘要，翻译为中文。材料如下：{str(info)}"
|
|
||||||
i_say_show_user = f'请你阅读以下学术论文相关的材料，提取摘要，翻译为中文。论文：{pdf_path}'
|
|
||||||
chatbot.append((i_say_show_user, "[Local Message] waiting gpt response."))
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
msg = '正常'
|
|
||||||
# ** gpt request **
|
|
||||||
# 单线，获取文章meta信息
|
|
||||||
gpt_say = yield from request_gpt_model_in_new_thread_with_ui_alive(
|
|
||||||
inputs=i_say,
|
|
||||||
inputs_show_user=i_say_show_user,
|
|
||||||
llm_kwargs=llm_kwargs,
|
|
||||||
chatbot=chatbot, history=[],
|
|
||||||
sys_prompt="Your job is to collect information from materials and translate to Chinese。",
|
|
||||||
)
|
|
||||||
|
|
||||||
chatbot[-1] = (i_say_show_user, gpt_say)
|
|
||||||
history.append(i_say_show_user); history.append(gpt_say)
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history, msg=msg) # 刷新界面
|
|
||||||
res = write_history_to_file(history)
|
|
||||||
promote_file_to_downloadzone(res, chatbot=chatbot)
|
|
||||||
promote_file_to_downloadzone(pdf_path, chatbot=chatbot)
|
|
||||||
|
|
||||||
chatbot.append(("完成了吗？", res + "\n\nPDF文件也已经下载"))
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history, msg=msg) # 刷新界面
|
|
||||||
|
|
||||||
@@ -1,40 +0,0 @@
|
|||||||
from toolbox import CatchException, update_ui, update_ui_lastest_msg
|
|
||||||
from crazy_functions.multi_stage.multi_stage_utils import GptAcademicGameBaseState
|
|
||||||
from crazy_functions.crazy_utils import request_gpt_model_in_new_thread_with_ui_alive
|
|
||||||
from request_llms.bridge_all import predict_no_ui_long_connection
|
|
||||||
from crazy_functions.game_fns.game_utils import get_code_block, is_same_thing
|
|
||||||
|
|
||||||
@CatchException
|
|
||||||
def 随机小游戏(prompt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_request):
|
|
||||||
from crazy_functions.game_fns.game_interactive_story import MiniGame_ResumeStory
|
|
||||||
# 清空历史
|
|
||||||
history = []
|
|
||||||
# 选择游戏
|
|
||||||
cls = MiniGame_ResumeStory
|
|
||||||
# 如果之前已经初始化了游戏实例，则继续该实例；否则重新初始化
|
|
||||||
state = cls.sync_state(chatbot,
|
|
||||||
llm_kwargs,
|
|
||||||
cls,
|
|
||||||
plugin_name='MiniGame_ResumeStory',
|
|
||||||
callback_fn='crazy_functions.互动小游戏->随机小游戏',
|
|
||||||
lock_plugin=True
|
|
||||||
)
|
|
||||||
yield from state.continue_game(prompt, chatbot, history)
|
|
||||||
|
|
||||||
|
|
||||||
@CatchException
|
|
||||||
def 随机小游戏1(prompt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_request):
|
|
||||||
from crazy_functions.game_fns.game_ascii_art import MiniGame_ASCII_Art
|
|
||||||
# 清空历史
|
|
||||||
history = []
|
|
||||||
# 选择游戏
|
|
||||||
cls = MiniGame_ASCII_Art
|
|
||||||
# 如果之前已经初始化了游戏实例，则继续该实例；否则重新初始化
|
|
||||||
state = cls.sync_state(chatbot,
|
|
||||||
llm_kwargs,
|
|
||||||
cls,
|
|
||||||
plugin_name='MiniGame_ASCII_Art',
|
|
||||||
callback_fn='crazy_functions.互动小游戏->随机小游戏1',
|
|
||||||
lock_plugin=True
|
|
||||||
)
|
|
||||||
yield from state.continue_game(prompt, chatbot, history)
|
|
||||||
@@ -1,62 +0,0 @@
|
|||||||
from toolbox import CatchException, update_ui
|
|
||||||
from crazy_functions.crazy_utils import request_gpt_model_in_new_thread_with_ui_alive
|
|
||||||
|
|
||||||
@CatchException
|
|
||||||
def 交互功能模板函数(txt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_request):
|
|
||||||
"""
|
|
||||||
txt 输入栏用户输入的文本，例如需要翻译的一段话，再例如一个包含了待处理文件的路径
|
|
||||||
llm_kwargs gpt模型参数, 如温度和top_p等, 一般原样传递下去就行
|
|
||||||
plugin_kwargs 插件模型的参数, 如温度和top_p等, 一般原样传递下去就行
|
|
||||||
chatbot 聊天显示框的句柄，用于显示给用户
|
|
||||||
history 聊天历史，前情提要
|
|
||||||
system_prompt 给gpt的静默提醒
|
|
||||||
user_request 当前用户的请求信息（IP地址等）
|
|
||||||
"""
|
|
||||||
history = [] # 清空历史，以免输入溢出
|
|
||||||
chatbot.append(("这是什么功能？", "交互功能函数模板。在执行完成之后, 可以将自身的状态存储到cookie中, 等待用户的再次调用。"))
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
|
|
||||||
state = chatbot._cookies.get('plugin_state_0001', None) # 初始化插件状态
|
|
||||||
|
|
||||||
if state is None:
|
|
||||||
chatbot._cookies['lock_plugin'] = 'crazy_functions.交互功能函数模板->交互功能模板函数' # 赋予插件锁定 锁定插件回调路径，当下一次用户提交时，会直接转到该函数
|
|
||||||
chatbot._cookies['plugin_state_0001'] = 'wait_user_keyword' # 赋予插件状态
|
|
||||||
|
|
||||||
chatbot.append(("第一次调用：", "请输入关键词, 我将为您查找相关壁纸, 建议使用英文单词, 插件锁定中，请直接提交即可。"))
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
|
|
||||||
if state == 'wait_user_keyword':
|
|
||||||
chatbot._cookies['lock_plugin'] = None # 解除插件锁定，避免遗忘导致死锁
|
|
||||||
chatbot._cookies['plugin_state_0001'] = None # 解除插件状态，避免遗忘导致死锁
|
|
||||||
|
|
||||||
# 解除插件锁定
|
|
||||||
chatbot.append((f"获取关键词：{txt}", ""))
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
page_return = get_image_page_by_keyword(txt)
|
|
||||||
inputs=inputs_show_user=f"Extract all image urls in this html page, pick the first 5 images and show them with markdown format: \n\n {page_return}"
|
|
||||||
gpt_say = yield from request_gpt_model_in_new_thread_with_ui_alive(
|
|
||||||
inputs=inputs, inputs_show_user=inputs_show_user,
|
|
||||||
llm_kwargs=llm_kwargs, chatbot=chatbot, history=[],
|
|
||||||
sys_prompt="When you want to show an image, use markdown format. e.g. . If there are no image url provided, answer 'no image url provided'"
|
|
||||||
)
|
|
||||||
chatbot[-1] = [chatbot[-1][0], gpt_say]
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------------
|
|
||||||
|
|
||||||
def get_image_page_by_keyword(keyword):
|
|
||||||
import requests
|
|
||||||
from bs4 import BeautifulSoup
|
|
||||||
response = requests.get(f'https://wallhaven.cc/search?q={keyword}', timeout=2)
|
|
||||||
res = "image urls: \n"
|
|
||||||
for image_element in BeautifulSoup(response.content, 'html.parser').findAll("img"):
|
|
||||||
try:
|
|
||||||
res += image_element["data-src"]
|
|
||||||
res += "\n"
|
|
||||||
except:
|
|
||||||
pass
|
|
||||||
return res
|
|
||||||
@@ -1,252 +0,0 @@
|
|||||||
# 本源代码中, ⭐ = 关键步骤
|
|
||||||
"""
|
|
||||||
测试：
|
|
||||||
- 裁剪图像，保留下半部分
|
|
||||||
- 交换图像的蓝色通道和红色通道
|
|
||||||
- 将图像转为灰度图像
|
|
||||||
- 将csv文件转excel表格
|
|
||||||
|
|
||||||
Testing:
|
|
||||||
- Crop the image, keeping the bottom half.
|
|
||||||
- Swap the blue channel and red channel of the image.
|
|
||||||
- Convert the image to grayscale.
|
|
||||||
- Convert the CSV file to an Excel spreadsheet.
|
|
||||||
"""
|
|
||||||
|
|
||||||
|
|
||||||
from toolbox import CatchException, update_ui, gen_time_str, trimmed_format_exc, is_the_upload_folder
|
|
||||||
from toolbox import promote_file_to_downloadzone, get_log_folder, update_ui_lastest_msg
|
|
||||||
from crazy_functions.crazy_utils import request_gpt_model_in_new_thread_with_ui_alive, get_plugin_arg
|
|
||||||
from crazy_functions.crazy_utils import input_clipping, try_install_deps
|
|
||||||
from crazy_functions.gen_fns.gen_fns_shared import is_function_successfully_generated
|
|
||||||
from crazy_functions.gen_fns.gen_fns_shared import get_class_name
|
|
||||||
from crazy_functions.gen_fns.gen_fns_shared import subprocess_worker
|
|
||||||
from crazy_functions.gen_fns.gen_fns_shared import try_make_module
|
|
||||||
import os
|
|
||||||
import time
|
|
||||||
import glob
|
|
||||||
import multiprocessing
|
|
||||||
|
|
||||||
templete = """
|
|
||||||
```python
|
|
||||||
import ... # Put dependencies here, e.g. import numpy as np.
|
|
||||||
|
|
||||||
class TerminalFunction(object): # Do not change the name of the class, The name of the class must be `TerminalFunction`
|
|
||||||
|
|
||||||
def run(self, path): # The name of the function must be `run`, it takes only a positional argument.
|
|
||||||
# rewrite the function you have just written here
|
|
||||||
...
|
|
||||||
return generated_file_path
|
|
||||||
```
|
|
||||||
"""
|
|
||||||
|
|
||||||
def inspect_dependency(chatbot, history):
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return True
|
|
||||||
|
|
||||||
def get_code_block(reply):
|
|
||||||
import re
|
|
||||||
pattern = r"```([\s\S]*?)```" # regex pattern to match code blocks
|
|
||||||
matches = re.findall(pattern, reply) # find all code blocks in text
|
|
||||||
if len(matches) == 1:
|
|
||||||
return matches[0].strip('python') # code block
|
|
||||||
for match in matches:
|
|
||||||
if 'class TerminalFunction' in match:
|
|
||||||
return match.strip('python') # code block
|
|
||||||
raise RuntimeError("GPT is not generating proper code.")
|
|
||||||
|
|
||||||
def gpt_interact_multi_step(txt, file_type, llm_kwargs, chatbot, history):
|
|
||||||
# 输入
|
|
||||||
prompt_compose = [
|
|
||||||
f'Your job:\n'
|
|
||||||
f'1. write a single Python function, which takes a path of a `{file_type}` file as the only argument and returns a `string` containing the result of analysis or the path of generated files. \n',
|
|
||||||
f"2. You should write this function to perform following task: " + txt + "\n",
|
|
||||||
f"3. Wrap the output python function with markdown codeblock."
|
|
||||||
]
|
|
||||||
i_say = "".join(prompt_compose)
|
|
||||||
demo = []
|
|
||||||
|
|
||||||
# 第一步
|
|
||||||
gpt_say = yield from request_gpt_model_in_new_thread_with_ui_alive(
|
|
||||||
inputs=i_say, inputs_show_user=i_say,
|
|
||||||
llm_kwargs=llm_kwargs, chatbot=chatbot, history=demo,
|
|
||||||
sys_prompt= r"You are a world-class programmer."
|
|
||||||
)
|
|
||||||
history.extend([i_say, gpt_say])
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面 # 界面更新
|
|
||||||
|
|
||||||
# 第二步
|
|
||||||
prompt_compose = [
|
|
||||||
"If previous stage is successful, rewrite the function you have just written to satisfy following templete: \n",
|
|
||||||
templete
|
|
||||||
]
|
|
||||||
i_say = "".join(prompt_compose); inputs_show_user = "If previous stage is successful, rewrite the function you have just written to satisfy executable templete. "
|
|
||||||
gpt_say = yield from request_gpt_model_in_new_thread_with_ui_alive(
|
|
||||||
inputs=i_say, inputs_show_user=inputs_show_user,
|
|
||||||
llm_kwargs=llm_kwargs, chatbot=chatbot, history=history,
|
|
||||||
sys_prompt= r"You are a programmer. You need to replace `...` with valid packages, do not give `...` in your answer!"
|
|
||||||
)
|
|
||||||
code_to_return = gpt_say
|
|
||||||
history.extend([i_say, gpt_say])
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面 # 界面更新
|
|
||||||
|
|
||||||
# # 第三步
|
|
||||||
# i_say = "Please list to packages to install to run the code above. Then show me how to use `try_install_deps` function to install them."
|
|
||||||
# i_say += 'For instance. `try_install_deps(["opencv-python", "scipy", "numpy"])`'
|
|
||||||
# installation_advance = yield from request_gpt_model_in_new_thread_with_ui_alive(
|
|
||||||
# inputs=i_say, inputs_show_user=inputs_show_user,
|
|
||||||
# llm_kwargs=llm_kwargs, chatbot=chatbot, history=history,
|
|
||||||
# sys_prompt= r"You are a programmer."
|
|
||||||
# )
|
|
||||||
|
|
||||||
# # # 第三步
|
|
||||||
# i_say = "Show me how to use `pip` to install packages to run the code above. "
|
|
||||||
# i_say += 'For instance. `pip install -r opencv-python scipy numpy`'
|
|
||||||
# installation_advance = yield from request_gpt_model_in_new_thread_with_ui_alive(
|
|
||||||
# inputs=i_say, inputs_show_user=i_say,
|
|
||||||
# llm_kwargs=llm_kwargs, chatbot=chatbot, history=history,
|
|
||||||
# sys_prompt= r"You are a programmer."
|
|
||||||
# )
|
|
||||||
installation_advance = ""
|
|
||||||
|
|
||||||
return code_to_return, installation_advance, txt, file_type, llm_kwargs, chatbot, history
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
def for_immediate_show_off_when_possible(file_type, fp, chatbot):
|
|
||||||
if file_type in ['png', 'jpg']:
|
|
||||||
image_path = os.path.abspath(fp)
|
|
||||||
chatbot.append(['这是一张图片, 展示如下:',
|
|
||||||
f'本地文件地址: <br/>`{image_path}`<br/>'+
|
|
||||||
f'本地文件预览: <br/><div align="center"><img src="file={image_path}"></div>'
|
|
||||||
])
|
|
||||||
return chatbot
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
def have_any_recent_upload_files(chatbot):
|
|
||||||
_5min = 5 * 60
|
|
||||||
if not chatbot: return False # chatbot is None
|
|
||||||
most_recent_uploaded = chatbot._cookies.get("most_recent_uploaded", None)
|
|
||||||
if not most_recent_uploaded: return False # most_recent_uploaded is None
|
|
||||||
if time.time() - most_recent_uploaded["time"] < _5min: return True # most_recent_uploaded is new
|
|
||||||
else: return False # most_recent_uploaded is too old
|
|
||||||
|
|
||||||
def get_recent_file_prompt_support(chatbot):
|
|
||||||
most_recent_uploaded = chatbot._cookies.get("most_recent_uploaded", None)
|
|
||||||
path = most_recent_uploaded['path']
|
|
||||||
return path
|
|
||||||
|
|
||||||
@CatchException
|
|
||||||
def 函数动态生成(txt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_request):
|
|
||||||
"""
|
|
||||||
txt 输入栏用户输入的文本，例如需要翻译的一段话，再例如一个包含了待处理文件的路径
|
|
||||||
llm_kwargs gpt模型参数，如温度和top_p等，一般原样传递下去就行
|
|
||||||
plugin_kwargs 插件模型的参数，暂时没有用武之地
|
|
||||||
chatbot 聊天显示框的句柄，用于显示给用户
|
|
||||||
history 聊天历史，前情提要
|
|
||||||
system_prompt 给gpt的静默提醒
|
|
||||||
user_request 当前用户的请求信息（IP地址等）
|
|
||||||
"""
|
|
||||||
|
|
||||||
# 清空历史
|
|
||||||
history = []
|
|
||||||
|
|
||||||
# 基本信息：功能、贡献者
|
|
||||||
chatbot.append(["正在启动: 插件动态生成插件", "插件动态生成, 执行开始, 作者Binary-Husky."])
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
|
|
||||||
# ⭐ 文件上传区是否有东西
|
|
||||||
# 1. 如果有文件: 作为函数参数
|
|
||||||
# 2. 如果没有文件：需要用GPT提取参数 （太懒了，以后再写，虚空终端已经实现了类似的代码）
|
|
||||||
file_list = []
|
|
||||||
if get_plugin_arg(plugin_kwargs, key="file_path_arg", default=False):
|
|
||||||
file_path = get_plugin_arg(plugin_kwargs, key="file_path_arg", default=None)
|
|
||||||
file_list.append(file_path)
|
|
||||||
yield from update_ui_lastest_msg(f"当前文件: {file_path}", chatbot, history, 1)
|
|
||||||
elif have_any_recent_upload_files(chatbot):
|
|
||||||
file_dir = get_recent_file_prompt_support(chatbot)
|
|
||||||
file_list = glob.glob(os.path.join(file_dir, '**/*'), recursive=True)
|
|
||||||
yield from update_ui_lastest_msg(f"当前文件处理列表: {file_list}", chatbot, history, 1)
|
|
||||||
else:
|
|
||||||
chatbot.append(["文件检索", "没有发现任何近期上传的文件。"])
|
|
||||||
yield from update_ui_lastest_msg("没有发现任何近期上传的文件。", chatbot, history, 1)
|
|
||||||
return # 2. 如果没有文件
|
|
||||||
if len(file_list) == 0:
|
|
||||||
chatbot.append(["文件检索", "没有发现任何近期上传的文件。"])
|
|
||||||
yield from update_ui_lastest_msg("没有发现任何近期上传的文件。", chatbot, history, 1)
|
|
||||||
return # 2. 如果没有文件
|
|
||||||
|
|
||||||
# 读取文件
|
|
||||||
file_type = file_list[0].split('.')[-1]
|
|
||||||
|
|
||||||
# 粗心检查
|
|
||||||
if is_the_upload_folder(txt):
|
|
||||||
yield from update_ui_lastest_msg(f"请在输入框内填写需求, 然后再次点击该插件! 至于您的文件，不用担心, 文件路径 {txt} 已经被记忆. ", chatbot, history, 1)
|
|
||||||
return
|
|
||||||
|
|
||||||
# 开始干正事
|
|
||||||
MAX_TRY = 3
|
|
||||||
for j in range(MAX_TRY): # 最多重试5次
|
|
||||||
traceback = ""
|
|
||||||
try:
|
|
||||||
# ⭐ 开始啦 ！
|
|
||||||
code, installation_advance, txt, file_type, llm_kwargs, chatbot, history = \
|
|
||||||
yield from gpt_interact_multi_step(txt, file_type, llm_kwargs, chatbot, history)
|
|
||||||
chatbot.append(["代码生成阶段结束", ""])
|
|
||||||
yield from update_ui_lastest_msg(f"正在验证上述代码的有效性 ...", chatbot, history, 1)
|
|
||||||
# ⭐ 分离代码块
|
|
||||||
code = get_code_block(code)
|
|
||||||
# ⭐ 检查模块
|
|
||||||
ok, traceback = try_make_module(code, chatbot)
|
|
||||||
# 搞定代码生成
|
|
||||||
if ok: break
|
|
||||||
except Exception as e:
|
|
||||||
if not traceback: traceback = trimmed_format_exc()
|
|
||||||
# 处理异常
|
|
||||||
if not traceback: traceback = trimmed_format_exc()
|
|
||||||
yield from update_ui_lastest_msg(f"第 {j+1}/{MAX_TRY} 次代码生成尝试, 失败了~ 别担心, 我们5秒后再试一次... \n\n此次我们的错误追踪是\n```\n{traceback}\n```\n", chatbot, history, 5)
|
|
||||||
|
|
||||||
# 代码生成结束, 开始执行
|
|
||||||
TIME_LIMIT = 15
|
|
||||||
yield from update_ui_lastest_msg(f"开始创建新进程并执行代码! 时间限制 {TIME_LIMIT} 秒. 请等待任务完成... ", chatbot, history, 1)
|
|
||||||
manager = multiprocessing.Manager()
|
|
||||||
return_dict = manager.dict()
|
|
||||||
|
|
||||||
# ⭐ 到最后一步了，开始逐个文件进行处理
|
|
||||||
for file_path in file_list:
|
|
||||||
if os.path.exists(file_path):
|
|
||||||
chatbot.append([f"正在处理文件: {file_path}", f"请稍等..."])
|
|
||||||
chatbot = for_immediate_show_off_when_possible(file_type, file_path, chatbot)
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面 # 界面更新
|
|
||||||
else:
|
|
||||||
continue
|
|
||||||
|
|
||||||
# ⭐⭐⭐ subprocess_worker ⭐⭐⭐
|
|
||||||
p = multiprocessing.Process(target=subprocess_worker, args=(code, file_path, return_dict))
|
|
||||||
# ⭐ 开始执行，时间限制TIME_LIMIT
|
|
||||||
p.start(); p.join(timeout=TIME_LIMIT)
|
|
||||||
if p.is_alive(): p.terminate(); p.join()
|
|
||||||
p.close()
|
|
||||||
res = return_dict['result']
|
|
||||||
success = return_dict['success']
|
|
||||||
traceback = return_dict['traceback']
|
|
||||||
if not success:
|
|
||||||
if not traceback: traceback = trimmed_format_exc()
|
|
||||||
chatbot.append(["执行失败了", f"错误追踪\n```\n{trimmed_format_exc()}\n```\n"])
|
|
||||||
# chatbot.append(["如果是缺乏依赖，请参考以下建议", installation_advance])
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
|
|
||||||
# 顺利完成，收尾
|
|
||||||
res = str(res)
|
|
||||||
if os.path.exists(res):
|
|
||||||
chatbot.append(["执行成功了，结果是一个有效文件", "结果：" + res])
|
|
||||||
new_file_path = promote_file_to_downloadzone(res, chatbot=chatbot)
|
|
||||||
chatbot = for_immediate_show_off_when_possible(file_type, new_file_path, chatbot)
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面 # 界面更新
|
|
||||||
else:
|
|
||||||
chatbot.append(["执行成功了，结果是一个字符串", "结果：" + res])
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面 # 界面更新
|
|
||||||
|
|
||||||
@@ -1,31 +0,0 @@
|
|||||||
from toolbox import CatchException, update_ui, gen_time_str
|
|
||||||
from crazy_functions.crazy_utils import request_gpt_model_in_new_thread_with_ui_alive
|
|
||||||
from crazy_functions.crazy_utils import input_clipping
|
|
||||||
import copy, json
|
|
||||||
|
|
||||||
@CatchException
|
|
||||||
def 命令行助手(txt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_request):
|
|
||||||
"""
|
|
||||||
txt 输入栏用户输入的文本, 例如需要翻译的一段话, 再例如一个包含了待处理文件的路径
|
|
||||||
llm_kwargs gpt模型参数, 如温度和top_p等, 一般原样传递下去就行
|
|
||||||
plugin_kwargs 插件模型的参数, 暂时没有用武之地
|
|
||||||
chatbot 聊天显示框的句柄, 用于显示给用户
|
|
||||||
history 聊天历史, 前情提要
|
|
||||||
system_prompt 给gpt的静默提醒
|
|
||||||
user_request 当前用户的请求信息（IP地址等）
|
|
||||||
"""
|
|
||||||
# 清空历史, 以免输入溢出
|
|
||||||
history = []
|
|
||||||
|
|
||||||
# 输入
|
|
||||||
i_say = "请写bash命令实现以下功能：" + txt
|
|
||||||
# 开始
|
|
||||||
gpt_say = yield from request_gpt_model_in_new_thread_with_ui_alive(
|
|
||||||
inputs=i_say, inputs_show_user=txt,
|
|
||||||
llm_kwargs=llm_kwargs, chatbot=chatbot, history=[],
|
|
||||||
sys_prompt="你是一个Linux大师级用户。注意，当我要求你写bash命令时，尽可能地仅用一行命令解决我的要求。"
|
|
||||||
)
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面 # 界面更新
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -1,102 +0,0 @@
|
|||||||
# 本源代码中, ⭐ = 关键步骤
|
|
||||||
"""
|
|
||||||
测试：
|
|
||||||
- show me the solution of $x^2=cos(x)$, solve this problem with figure, and plot and save image to t.jpg
|
|
||||||
|
|
||||||
"""
|
|
||||||
|
|
||||||
|
|
||||||
import time
|
|
||||||
from toolbox import CatchException, update_ui, gen_time_str, trimmed_format_exc, ProxyNetworkActivate
|
|
||||||
from toolbox import get_conf, select_api_key, update_ui_lastest_msg, Singleton
|
|
||||||
from crazy_functions.crazy_utils import request_gpt_model_in_new_thread_with_ui_alive, get_plugin_arg
|
|
||||||
from crazy_functions.crazy_utils import input_clipping, try_install_deps
|
|
||||||
from crazy_functions.agent_fns.persistent import GradioMultiuserManagerForPersistentClasses
|
|
||||||
from crazy_functions.agent_fns.auto_agent import AutoGenMath
|
|
||||||
from loguru import logger
|
|
||||||
|
|
||||||
def remove_model_prefix(llm):
|
|
||||||
if llm.startswith('api2d-'): llm = llm.replace('api2d-', '')
|
|
||||||
if llm.startswith('azure-'): llm = llm.replace('azure-', '')
|
|
||||||
return llm
|
|
||||||
|
|
||||||
|
|
||||||
@CatchException
|
|
||||||
def 多智能体终端(txt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_request):
|
|
||||||
"""
|
|
||||||
txt 输入栏用户输入的文本，例如需要翻译的一段话，再例如一个包含了待处理文件的路径
|
|
||||||
llm_kwargs gpt模型参数，如温度和top_p等，一般原样传递下去就行
|
|
||||||
plugin_kwargs 插件模型的参数
|
|
||||||
chatbot 聊天显示框的句柄，用于显示给用户
|
|
||||||
history 聊天历史，前情提要
|
|
||||||
system_prompt 给gpt的静默提醒
|
|
||||||
user_request 当前用户的请求信息（IP地址等）
|
|
||||||
"""
|
|
||||||
# 检查当前的模型是否符合要求
|
|
||||||
supported_llms = [
|
|
||||||
"gpt-3.5-turbo-16k",
|
|
||||||
'gpt-3.5-turbo-1106',
|
|
||||||
"gpt-4",
|
|
||||||
"gpt-4-32k",
|
|
||||||
'gpt-4-1106-preview',
|
|
||||||
"azure-gpt-3.5-turbo-16k",
|
|
||||||
"azure-gpt-3.5-16k",
|
|
||||||
"azure-gpt-4",
|
|
||||||
"azure-gpt-4-32k",
|
|
||||||
]
|
|
||||||
from request_llms.bridge_all import model_info
|
|
||||||
if model_info[llm_kwargs['llm_model']]["max_token"] < 8000: # 至少是8k上下文的模型
|
|
||||||
chatbot.append([f"处理任务: {txt}", f"当前插件只支持{str(supported_llms)}, 当前模型{llm_kwargs['llm_model']}的最大上下文长度太短, 不能支撑AutoGen运行。"])
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
if model_info[llm_kwargs['llm_model']]["endpoint"] is not None: # 如果不是本地模型，加载API_KEY
|
|
||||||
llm_kwargs['api_key'] = select_api_key(llm_kwargs['api_key'], llm_kwargs['llm_model'])
|
|
||||||
|
|
||||||
# 尝试导入依赖，如果缺少依赖，则给出安装建议
|
|
||||||
try:
|
|
||||||
import autogen
|
|
||||||
if get_conf("AUTOGEN_USE_DOCKER"):
|
|
||||||
import docker
|
|
||||||
except:
|
|
||||||
chatbot.append([ f"处理任务: {txt}",
|
|
||||||
f"导入软件依赖失败。使用该模块需要额外依赖，安装方法```pip install --upgrade pyautogen docker```。"])
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
|
|
||||||
# 尝试导入依赖，如果缺少依赖，则给出安装建议
|
|
||||||
try:
|
|
||||||
import autogen
|
|
||||||
import glob, os, time, subprocess
|
|
||||||
if get_conf("AUTOGEN_USE_DOCKER"):
|
|
||||||
subprocess.Popen(["docker", "--version"])
|
|
||||||
except:
|
|
||||||
chatbot.append([f"处理任务: {txt}", f"缺少docker运行环境！"])
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
|
|
||||||
# 解锁插件
|
|
||||||
chatbot.get_cookies()['lock_plugin'] = None
|
|
||||||
persistent_class_multi_user_manager = GradioMultiuserManagerForPersistentClasses()
|
|
||||||
user_uuid = chatbot.get_cookies().get('uuid')
|
|
||||||
persistent_key = f"{user_uuid}->多智能体终端"
|
|
||||||
if persistent_class_multi_user_manager.already_alive(persistent_key):
|
|
||||||
# 当已经存在一个正在运行的多智能体终端时，直接将用户输入传递给它，而不是再次启动一个新的多智能体终端
|
|
||||||
logger.info('[debug] feed new user input')
|
|
||||||
executor = persistent_class_multi_user_manager.get(persistent_key)
|
|
||||||
exit_reason = yield from executor.main_process_ui_control(txt, create_or_resume="resume")
|
|
||||||
else:
|
|
||||||
# 运行多智能体终端 (首次)
|
|
||||||
logger.info('[debug] create new executor instance')
|
|
||||||
history = []
|
|
||||||
chatbot.append(["正在启动: 多智能体终端", "插件动态生成, 执行开始, 作者 Microsoft & Binary-Husky."])
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
executor = AutoGenMath(llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_request)
|
|
||||||
persistent_class_multi_user_manager.set(persistent_key, executor)
|
|
||||||
exit_reason = yield from executor.main_process_ui_control(txt, create_or_resume="create")
|
|
||||||
|
|
||||||
if exit_reason == "wait_feedback":
|
|
||||||
# 当用户点击了“等待反馈”按钮时，将executor存储到cookie中，等待用户的再次调用
|
|
||||||
executor.chatbot.get_cookies()['lock_plugin'] = 'crazy_functions.多智能体->多智能体终端'
|
|
||||||
else:
|
|
||||||
executor.chatbot.get_cookies()['lock_plugin'] = None
|
|
||||||
yield from update_ui(chatbot=executor.chatbot, history=executor.history) # 更新状态
|
|
||||||
@@ -1,127 +0,0 @@
|
|||||||
from toolbox import update_ui
|
|
||||||
from toolbox import CatchException, report_exception
|
|
||||||
from toolbox import write_history_to_file, promote_file_to_downloadzone
|
|
||||||
from crazy_functions.crazy_utils import request_gpt_model_in_new_thread_with_ui_alive
|
|
||||||
fast_debug = False
|
|
||||||
|
|
||||||
|
|
||||||
def 解析docx(file_manifest, project_folder, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt):
|
|
||||||
import time, os
|
|
||||||
# pip install python-docx 用于docx格式，跨平台
|
|
||||||
# pip install pywin32 用于doc格式，仅支持Win平台
|
|
||||||
for index, fp in enumerate(file_manifest):
|
|
||||||
if fp.split(".")[-1] == "docx":
|
|
||||||
from docx import Document
|
|
||||||
doc = Document(fp)
|
|
||||||
file_content = "\n".join([para.text for para in doc.paragraphs])
|
|
||||||
else:
|
|
||||||
try:
|
|
||||||
import win32com.client
|
|
||||||
word = win32com.client.Dispatch("Word.Application")
|
|
||||||
word.visible = False
|
|
||||||
# 打开文件
|
|
||||||
doc = word.Documents.Open(os.getcwd() + '/' + fp)
|
|
||||||
# file_content = doc.Content.Text
|
|
||||||
doc = word.ActiveDocument
|
|
||||||
file_content = doc.Range().Text
|
|
||||||
doc.Close()
|
|
||||||
word.Quit()
|
|
||||||
except:
|
|
||||||
raise RuntimeError('请先将.doc文档转换为.docx文档。')
|
|
||||||
|
|
||||||
# private_upload里面的文件名在解压zip后容易出现乱码（rar和7z格式正常），故可以只分析文章内容，不输入文件名
|
|
||||||
from crazy_functions.pdf_fns.breakdown_txt import breakdown_text_to_satisfy_token_limit
|
|
||||||
from request_llms.bridge_all import model_info
|
|
||||||
max_token = model_info[llm_kwargs['llm_model']]['max_token']
|
|
||||||
TOKEN_LIMIT_PER_FRAGMENT = max_token * 3 // 4
|
|
||||||
paper_fragments = breakdown_text_to_satisfy_token_limit(txt=file_content, limit=TOKEN_LIMIT_PER_FRAGMENT, llm_model=llm_kwargs['llm_model'])
|
|
||||||
this_paper_history = []
|
|
||||||
for i, paper_frag in enumerate(paper_fragments):
|
|
||||||
i_say = f'请对下面的文章片段用中文做概述，文件名是{os.path.relpath(fp, project_folder)}，文章内容是 ```{paper_frag}```'
|
|
||||||
i_say_show_user = f'请对下面的文章片段做概述: {os.path.abspath(fp)}的第{i+1}/{len(paper_fragments)}个片段。'
|
|
||||||
gpt_say = yield from request_gpt_model_in_new_thread_with_ui_alive(
|
|
||||||
inputs=i_say,
|
|
||||||
inputs_show_user=i_say_show_user,
|
|
||||||
llm_kwargs=llm_kwargs,
|
|
||||||
chatbot=chatbot,
|
|
||||||
history=[],
|
|
||||||
sys_prompt="总结文章。"
|
|
||||||
)
|
|
||||||
|
|
||||||
chatbot[-1] = (i_say_show_user, gpt_say)
|
|
||||||
history.extend([i_say_show_user,gpt_say])
|
|
||||||
this_paper_history.extend([i_say_show_user,gpt_say])
|
|
||||||
|
|
||||||
# 已经对该文章的所有片段总结完毕，如果文章被切分了，
|
|
||||||
if len(paper_fragments) > 1:
|
|
||||||
i_say = f"根据以上的对话，总结文章{os.path.abspath(fp)}的主要内容。"
|
|
||||||
gpt_say = yield from request_gpt_model_in_new_thread_with_ui_alive(
|
|
||||||
inputs=i_say,
|
|
||||||
inputs_show_user=i_say,
|
|
||||||
llm_kwargs=llm_kwargs,
|
|
||||||
chatbot=chatbot,
|
|
||||||
history=this_paper_history,
|
|
||||||
sys_prompt="总结文章。"
|
|
||||||
)
|
|
||||||
|
|
||||||
history.extend([i_say,gpt_say])
|
|
||||||
this_paper_history.extend([i_say,gpt_say])
|
|
||||||
|
|
||||||
res = write_history_to_file(history)
|
|
||||||
promote_file_to_downloadzone(res, chatbot=chatbot)
|
|
||||||
chatbot.append(("完成了吗？", res))
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
|
|
||||||
res = write_history_to_file(history)
|
|
||||||
promote_file_to_downloadzone(res, chatbot=chatbot)
|
|
||||||
chatbot.append(("所有文件都总结完成了吗？", res))
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
|
|
||||||
|
|
||||||
@CatchException
|
|
||||||
def 总结word文档(txt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_request):
|
|
||||||
import glob, os
|
|
||||||
|
|
||||||
# 基本信息：功能、贡献者
|
|
||||||
chatbot.append([
|
|
||||||
"函数插件功能？",
|
|
||||||
"批量总结Word文档。函数插件贡献者: JasonGuo1。注意, 如果是.doc文件, 请先转化为.docx格式。"])
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
|
|
||||||
# 尝试导入依赖，如果缺少依赖，则给出安装建议
|
|
||||||
try:
|
|
||||||
from docx import Document
|
|
||||||
except:
|
|
||||||
report_exception(chatbot, history,
|
|
||||||
a=f"解析项目: {txt}",
|
|
||||||
b=f"导入软件依赖失败。使用该模块需要额外依赖，安装方法```pip install --upgrade python-docx pywin32```。")
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
|
|
||||||
# 清空历史，以免输入溢出
|
|
||||||
history = []
|
|
||||||
|
|
||||||
# 检测输入参数，如没有给定输入参数，直接退出
|
|
||||||
if os.path.exists(txt):
|
|
||||||
project_folder = txt
|
|
||||||
else:
|
|
||||||
if txt == "": txt = '空空如也的输入栏'
|
|
||||||
report_exception(chatbot, history, a=f"解析项目: {txt}", b=f"找不到本地项目或无权访问: {txt}")
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
|
|
||||||
# 搜索需要处理的文件清单
|
|
||||||
if txt.endswith('.docx') or txt.endswith('.doc'):
|
|
||||||
file_manifest = [txt]
|
|
||||||
else:
|
|
||||||
file_manifest = [f for f in glob.glob(f'{project_folder}/**/*.docx', recursive=True)] + \
|
|
||||||
[f for f in glob.glob(f'{project_folder}/**/*.doc', recursive=True)]
|
|
||||||
|
|
||||||
# 如果没找到任何文件
|
|
||||||
if len(file_manifest) == 0:
|
|
||||||
report_exception(chatbot, history, a=f"解析项目: {txt}", b=f"找不到任何.docx或doc文件: {txt}")
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
|
|
||||||
# 开始正式执行任务
|
|
||||||
yield from 解析docx(file_manifest, project_folder, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt)
|
|
||||||
@@ -1,186 +0,0 @@
|
|||||||
from toolbox import CatchException, report_exception, select_api_key, update_ui, get_conf
|
|
||||||
from crazy_functions.crazy_utils import request_gpt_model_in_new_thread_with_ui_alive
|
|
||||||
from toolbox import write_history_to_file, promote_file_to_downloadzone, get_log_folder
|
|
||||||
|
|
||||||
def split_audio_file(filename, split_duration=1000):
|
|
||||||
"""
|
|
||||||
根据给定的切割时长将音频文件切割成多个片段。
|
|
||||||
|
|
||||||
Args:
|
|
||||||
filename (str): 需要被切割的音频文件名。
|
|
||||||
split_duration (int, optional): 每个切割音频片段的时长（以秒为单位）。默认值为1000。
|
|
||||||
|
|
||||||
Returns:
|
|
||||||
filelist (list): 一个包含所有切割音频片段文件路径的列表。
|
|
||||||
|
|
||||||
"""
|
|
||||||
from moviepy.editor import AudioFileClip
|
|
||||||
import os
|
|
||||||
os.makedirs(f"{get_log_folder(plugin_name='audio')}/mp3/cut/", exist_ok=True) # 创建存储切割音频的文件夹
|
|
||||||
|
|
||||||
# 读取音频文件
|
|
||||||
audio = AudioFileClip(filename)
|
|
||||||
|
|
||||||
# 计算文件总时长和切割点
|
|
||||||
total_duration = audio.duration
|
|
||||||
split_points = list(range(0, int(total_duration), split_duration))
|
|
||||||
split_points.append(int(total_duration))
|
|
||||||
filelist = []
|
|
||||||
|
|
||||||
# 切割音频文件
|
|
||||||
for i in range(len(split_points) - 1):
|
|
||||||
start_time = split_points[i]
|
|
||||||
end_time = split_points[i + 1]
|
|
||||||
split_audio = audio.subclip(start_time, end_time)
|
|
||||||
split_audio.write_audiofile(f"{get_log_folder(plugin_name='audio')}/mp3/cut/{filename[0]}_{i}.mp3")
|
|
||||||
filelist.append(f"{get_log_folder(plugin_name='audio')}/mp3/cut/{filename[0]}_{i}.mp3")
|
|
||||||
|
|
||||||
audio.close()
|
|
||||||
return filelist
|
|
||||||
|
|
||||||
def AnalyAudio(parse_prompt, file_manifest, llm_kwargs, chatbot, history):
|
|
||||||
import os, requests
|
|
||||||
from moviepy.editor import AudioFileClip
|
|
||||||
from request_llms.bridge_all import model_info
|
|
||||||
|
|
||||||
# 设置OpenAI密钥和模型
|
|
||||||
api_key = select_api_key(llm_kwargs['api_key'], llm_kwargs['llm_model'])
|
|
||||||
chat_endpoint = model_info[llm_kwargs['llm_model']]['endpoint']
|
|
||||||
|
|
||||||
whisper_endpoint = chat_endpoint.replace('chat/completions', 'audio/transcriptions')
|
|
||||||
url = whisper_endpoint
|
|
||||||
headers = {
|
|
||||||
'Authorization': f"Bearer {api_key}"
|
|
||||||
}
|
|
||||||
|
|
||||||
os.makedirs(f"{get_log_folder(plugin_name='audio')}/mp3/", exist_ok=True)
|
|
||||||
for index, fp in enumerate(file_manifest):
|
|
||||||
audio_history = []
|
|
||||||
# 提取文件扩展名
|
|
||||||
ext = os.path.splitext(fp)[1]
|
|
||||||
# 提取视频中的音频
|
|
||||||
if ext not in [".mp3", ".wav", ".m4a", ".mpga"]:
|
|
||||||
audio_clip = AudioFileClip(fp)
|
|
||||||
audio_clip.write_audiofile(f"{get_log_folder(plugin_name='audio')}/mp3/output{index}.mp3")
|
|
||||||
fp = f"{get_log_folder(plugin_name='audio')}/mp3/output{index}.mp3"
|
|
||||||
# 调用whisper模型音频转文字
|
|
||||||
voice = split_audio_file(fp)
|
|
||||||
for j, i in enumerate(voice):
|
|
||||||
with open(i, 'rb') as f:
|
|
||||||
file_content = f.read() # 读取文件内容到内存
|
|
||||||
files = {
|
|
||||||
'file': (os.path.basename(i), file_content),
|
|
||||||
}
|
|
||||||
data = {
|
|
||||||
"model": "whisper-1",
|
|
||||||
"prompt": parse_prompt,
|
|
||||||
'response_format': "text"
|
|
||||||
}
|
|
||||||
|
|
||||||
chatbot.append([f"将 {i} 发送到openai音频解析终端 (whisper)，当前参数：{parse_prompt}", "正在处理 ..."])
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
proxies = get_conf('proxies')
|
|
||||||
response = requests.post(url, headers=headers, files=files, data=data, proxies=proxies).text
|
|
||||||
|
|
||||||
chatbot.append(["音频解析结果", response])
|
|
||||||
history.extend(["音频解析结果", response])
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
|
|
||||||
i_say = f'请对下面的音频片段做概述，音频内容是 ```{response}```'
|
|
||||||
i_say_show_user = f'第{index + 1}段音频的第{j + 1} / {len(voice)}片段。'
|
|
||||||
gpt_say = yield from request_gpt_model_in_new_thread_with_ui_alive(
|
|
||||||
inputs=i_say,
|
|
||||||
inputs_show_user=i_say_show_user,
|
|
||||||
llm_kwargs=llm_kwargs,
|
|
||||||
chatbot=chatbot,
|
|
||||||
history=[],
|
|
||||||
sys_prompt=f"总结音频。音频文件名{fp}"
|
|
||||||
)
|
|
||||||
|
|
||||||
chatbot[-1] = (i_say_show_user, gpt_say)
|
|
||||||
history.extend([i_say_show_user, gpt_say])
|
|
||||||
audio_history.extend([i_say_show_user, gpt_say])
|
|
||||||
|
|
||||||
# 已经对该文章的所有片段总结完毕，如果文章被切分了
|
|
||||||
result = "".join(audio_history)
|
|
||||||
if len(audio_history) > 1:
|
|
||||||
i_say = f"根据以上的对话，使用中文总结音频“{result}”的主要内容。"
|
|
||||||
i_say_show_user = f'第{index + 1}段音频的主要内容：'
|
|
||||||
gpt_say = yield from request_gpt_model_in_new_thread_with_ui_alive(
|
|
||||||
inputs=i_say,
|
|
||||||
inputs_show_user=i_say_show_user,
|
|
||||||
llm_kwargs=llm_kwargs,
|
|
||||||
chatbot=chatbot,
|
|
||||||
history=audio_history,
|
|
||||||
sys_prompt="总结文章。"
|
|
||||||
)
|
|
||||||
history.extend([i_say, gpt_say])
|
|
||||||
audio_history.extend([i_say, gpt_say])
|
|
||||||
|
|
||||||
res = write_history_to_file(history)
|
|
||||||
promote_file_to_downloadzone(res, chatbot=chatbot)
|
|
||||||
chatbot.append((f"第{index + 1}段音频完成了吗？", res))
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
|
|
||||||
# 删除中间文件夹
|
|
||||||
import shutil
|
|
||||||
shutil.rmtree(f"{get_log_folder(plugin_name='audio')}/mp3")
|
|
||||||
res = write_history_to_file(history)
|
|
||||||
promote_file_to_downloadzone(res, chatbot=chatbot)
|
|
||||||
chatbot.append(("所有音频都总结完成了吗？", res))
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history)
|
|
||||||
|
|
||||||
|
|
||||||
@CatchException
|
|
||||||
def 总结音视频(txt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, WEB_PORT):
|
|
||||||
import glob, os
|
|
||||||
|
|
||||||
# 基本信息：功能、贡献者
|
|
||||||
chatbot.append([
|
|
||||||
"函数插件功能？",
|
|
||||||
"总结音视频内容，函数插件贡献者: dalvqw & BinaryHusky"])
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
|
|
||||||
try:
|
|
||||||
from moviepy.editor import AudioFileClip
|
|
||||||
except:
|
|
||||||
report_exception(chatbot, history,
|
|
||||||
a=f"解析项目: {txt}",
|
|
||||||
b=f"导入软件依赖失败。使用该模块需要额外依赖，安装方法```pip install --upgrade moviepy```。")
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
|
|
||||||
# 清空历史，以免输入溢出
|
|
||||||
history = []
|
|
||||||
|
|
||||||
# 检测输入参数，如没有给定输入参数，直接退出
|
|
||||||
if os.path.exists(txt):
|
|
||||||
project_folder = txt
|
|
||||||
else:
|
|
||||||
if txt == "": txt = '空空如也的输入栏'
|
|
||||||
report_exception(chatbot, history, a=f"解析项目: {txt}", b=f"找不到本地项目或无权访问: {txt}")
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
|
|
||||||
# 搜索需要处理的文件清单
|
|
||||||
extensions = ['.mp4', '.m4a', '.wav', '.mpga', '.mpeg', '.mp3', '.avi', '.mkv', '.flac', '.aac']
|
|
||||||
|
|
||||||
if txt.endswith(tuple(extensions)):
|
|
||||||
file_manifest = [txt]
|
|
||||||
else:
|
|
||||||
file_manifest = []
|
|
||||||
for extension in extensions:
|
|
||||||
file_manifest.extend(glob.glob(f'{project_folder}/**/*{extension}', recursive=True))
|
|
||||||
|
|
||||||
# 如果没找到任何文件
|
|
||||||
if len(file_manifest) == 0:
|
|
||||||
report_exception(chatbot, history, a=f"解析项目: {txt}", b=f"找不到任何音频或视频文件: {txt}")
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
|
|
||||||
# 开始正式执行任务
|
|
||||||
if ("advanced_arg" in plugin_kwargs) and (plugin_kwargs["advanced_arg"] == ""): plugin_kwargs.pop("advanced_arg")
|
|
||||||
parse_prompt = plugin_kwargs.get("advanced_arg", '将音频解析为简体中文')
|
|
||||||
yield from AnalyAudio(parse_prompt, file_manifest, llm_kwargs, chatbot, history)
|
|
||||||
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
@@ -1,147 +0,0 @@
|
|||||||
from loguru import logger
|
|
||||||
|
|
||||||
from toolbox import update_ui, promote_file_to_downloadzone, gen_time_str
|
|
||||||
from toolbox import CatchException, report_exception
|
|
||||||
from toolbox import write_history_to_file, promote_file_to_downloadzone
|
|
||||||
from crazy_functions.crazy_utils import request_gpt_model_in_new_thread_with_ui_alive
|
|
||||||
from crazy_functions.crazy_utils import read_and_clean_pdf_text
|
|
||||||
from crazy_functions.crazy_utils import input_clipping
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
def 解析PDF(file_manifest, project_folder, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt):
|
|
||||||
file_write_buffer = []
|
|
||||||
for file_name in file_manifest:
|
|
||||||
logger.info('begin analysis on:', file_name)
|
|
||||||
############################## <第 0 步，切割PDF> ##################################
|
|
||||||
# 递归地切割PDF文件，每一块（尽量是完整的一个section，比如introduction，experiment等，必要时再进行切割）
|
|
||||||
# 的长度必须小于 2500 个 Token
|
|
||||||
file_content, page_one = read_and_clean_pdf_text(file_name) # （尝试）按照章节切割PDF
|
|
||||||
file_content = file_content.encode('utf-8', 'ignore').decode() # avoid reading non-utf8 chars
|
|
||||||
page_one = str(page_one).encode('utf-8', 'ignore').decode() # avoid reading non-utf8 chars
|
|
||||||
|
|
||||||
TOKEN_LIMIT_PER_FRAGMENT = 2500
|
|
||||||
|
|
||||||
from crazy_functions.pdf_fns.breakdown_txt import breakdown_text_to_satisfy_token_limit
|
|
||||||
paper_fragments = breakdown_text_to_satisfy_token_limit(txt=file_content, limit=TOKEN_LIMIT_PER_FRAGMENT, llm_model=llm_kwargs['llm_model'])
|
|
||||||
page_one_fragments = breakdown_text_to_satisfy_token_limit(txt=str(page_one), limit=TOKEN_LIMIT_PER_FRAGMENT//4, llm_model=llm_kwargs['llm_model'])
|
|
||||||
# 为了更好的效果，我们剥离Introduction之后的部分（如果有）
|
|
||||||
paper_meta = page_one_fragments[0].split('introduction')[0].split('Introduction')[0].split('INTRODUCTION')[0]
|
|
||||||
|
|
||||||
############################## <第 1 步，从摘要中提取高价值信息，放到history中> ##################################
|
|
||||||
final_results = []
|
|
||||||
final_results.append(paper_meta)
|
|
||||||
|
|
||||||
############################## <第 2 步，迭代地历遍整个文章，提取精炼信息> ##################################
|
|
||||||
i_say_show_user = f'首先你在中文语境下通读整篇论文。'; gpt_say = "[Local Message] 收到。" # 用户提示
|
|
||||||
chatbot.append([i_say_show_user, gpt_say]); yield from update_ui(chatbot=chatbot, history=[]) # 更新UI
|
|
||||||
|
|
||||||
iteration_results = []
|
|
||||||
last_iteration_result = paper_meta # 初始值是摘要
|
|
||||||
MAX_WORD_TOTAL = 4096 * 0.7
|
|
||||||
n_fragment = len(paper_fragments)
|
|
||||||
if n_fragment >= 20: logger.warning('文章极长，不能达到预期效果')
|
|
||||||
for i in range(n_fragment):
|
|
||||||
NUM_OF_WORD = MAX_WORD_TOTAL // n_fragment
|
|
||||||
i_say = f"Read this section, recapitulate the content of this section with less than {NUM_OF_WORD} Chinese characters: {paper_fragments[i]}"
|
|
||||||
i_say_show_user = f"[{i+1}/{n_fragment}] Read this section, recapitulate the content of this section with less than {NUM_OF_WORD} Chinese characters: {paper_fragments[i][:200]}"
|
|
||||||
gpt_say = yield from request_gpt_model_in_new_thread_with_ui_alive(i_say, i_say_show_user, # i_say=真正给chatgpt的提问， i_say_show_user=给用户看的提问
|
|
||||||
llm_kwargs, chatbot,
|
|
||||||
history=["The main idea of the previous section is?", last_iteration_result], # 迭代上一次的结果
|
|
||||||
sys_prompt="Extract the main idea of this section with Chinese." # 提示
|
|
||||||
)
|
|
||||||
iteration_results.append(gpt_say)
|
|
||||||
last_iteration_result = gpt_say
|
|
||||||
|
|
||||||
############################## <第 3 步，整理history，提取总结> ##################################
|
|
||||||
final_results.extend(iteration_results)
|
|
||||||
final_results.append(f'Please conclude this paper discussed above。')
|
|
||||||
# This prompt is from https://github.com/kaixindelele/ChatPaper/blob/main/chat_paper.py
|
|
||||||
NUM_OF_WORD = 1000
|
|
||||||
i_say = """
|
|
||||||
1. Mark the title of the paper (with Chinese translation)
|
|
||||||
2. list all the authors' names (use English)
|
|
||||||
3. mark the first author's affiliation (output Chinese translation only)
|
|
||||||
4. mark the keywords of this article (use English)
|
|
||||||
5. link to the paper, Github code link (if available, fill in Github:None if not)
|
|
||||||
6. summarize according to the following four points.Be sure to use Chinese answers (proper nouns need to be marked in English)
|
|
||||||
- (1):What is the research background of this article?
|
|
||||||
- (2):What are the past methods? What are the problems with them? Is the approach well motivated?
|
|
||||||
- (3):What is the research methodology proposed in this paper?
|
|
||||||
- (4):On what task and what performance is achieved by the methods in this paper? Can the performance support their goals?
|
|
||||||
Follow the format of the output that follows:
|
|
||||||
1. Title: xxx\n\n
|
|
||||||
2. Authors: xxx\n\n
|
|
||||||
3. Affiliation: xxx\n\n
|
|
||||||
4. Keywords: xxx\n\n
|
|
||||||
5. Urls: xxx or xxx , xxx \n\n
|
|
||||||
6. Summary: \n\n
|
|
||||||
- (1):xxx;\n
|
|
||||||
- (2):xxx;\n
|
|
||||||
- (3):xxx;\n
|
|
||||||
- (4):xxx.\n\n
|
|
||||||
Be sure to use Chinese answers (proper nouns need to be marked in English), statements as concise and academic as possible,
|
|
||||||
do not have too much repetitive information, numerical values using the original numbers.
|
|
||||||
"""
|
|
||||||
# This prompt is from https://github.com/kaixindelele/ChatPaper/blob/main/chat_paper.py
|
|
||||||
file_write_buffer.extend(final_results)
|
|
||||||
i_say, final_results = input_clipping(i_say, final_results, max_token_limit=2000)
|
|
||||||
gpt_say = yield from request_gpt_model_in_new_thread_with_ui_alive(
|
|
||||||
inputs=i_say, inputs_show_user='开始最终总结',
|
|
||||||
llm_kwargs=llm_kwargs, chatbot=chatbot, history=final_results,
|
|
||||||
sys_prompt= f"Extract the main idea of this paper with less than {NUM_OF_WORD} Chinese characters"
|
|
||||||
)
|
|
||||||
final_results.append(gpt_say)
|
|
||||||
file_write_buffer.extend([i_say, gpt_say])
|
|
||||||
############################## <第 4 步，设置一个token上限> ##################################
|
|
||||||
_, final_results = input_clipping("", final_results, max_token_limit=3200)
|
|
||||||
yield from update_ui(chatbot=chatbot, history=final_results) # 注意这里的历史记录被替代了
|
|
||||||
|
|
||||||
res = write_history_to_file(file_write_buffer)
|
|
||||||
promote_file_to_downloadzone(res, chatbot=chatbot)
|
|
||||||
yield from update_ui(chatbot=chatbot, history=final_results) # 刷新界面
|
|
||||||
|
|
||||||
|
|
||||||
@CatchException
|
|
||||||
def 批量总结PDF文档(txt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_request):
|
|
||||||
import glob, os
|
|
||||||
|
|
||||||
# 基本信息：功能、贡献者
|
|
||||||
chatbot.append([
|
|
||||||
"函数插件功能？",
|
|
||||||
"批量总结PDF文档。函数插件贡献者: ValeriaWong，Eralien"])
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
|
|
||||||
# 尝试导入依赖，如果缺少依赖，则给出安装建议
|
|
||||||
try:
|
|
||||||
import fitz
|
|
||||||
except:
|
|
||||||
report_exception(chatbot, history,
|
|
||||||
a = f"解析项目: {txt}",
|
|
||||||
b = f"导入软件依赖失败。使用该模块需要额外依赖，安装方法```pip install --upgrade pymupdf```。")
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
|
|
||||||
# 清空历史，以免输入溢出
|
|
||||||
history = []
|
|
||||||
|
|
||||||
# 检测输入参数，如没有给定输入参数，直接退出
|
|
||||||
if os.path.exists(txt):
|
|
||||||
project_folder = txt
|
|
||||||
else:
|
|
||||||
if txt == "": txt = '空空如也的输入栏'
|
|
||||||
report_exception(chatbot, history, a = f"解析项目: {txt}", b = f"找不到本地项目或无权访问: {txt}")
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
|
|
||||||
# 搜索需要处理的文件清单
|
|
||||||
file_manifest = [f for f in glob.glob(f'{project_folder}/**/*.pdf', recursive=True)]
|
|
||||||
|
|
||||||
# 如果没找到任何文件
|
|
||||||
if len(file_manifest) == 0:
|
|
||||||
report_exception(chatbot, history, a = f"解析项目: {txt}", b = f"找不到任何.tex或.pdf文件: {txt}")
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
|
|
||||||
# 开始正式执行任务
|
|
||||||
yield from 解析PDF(file_manifest, project_folder, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt)
|
|
||||||
@@ -1,162 +0,0 @@
|
|||||||
from loguru import logger
|
|
||||||
from toolbox import update_ui
|
|
||||||
from toolbox import CatchException, report_exception
|
|
||||||
from crazy_functions.crazy_utils import request_gpt_model_in_new_thread_with_ui_alive
|
|
||||||
from toolbox import write_history_to_file, promote_file_to_downloadzone
|
|
||||||
|
|
||||||
fast_debug = False
|
|
||||||
|
|
||||||
def readPdf(pdfPath):
|
|
||||||
"""
|
|
||||||
读取pdf文件，返回文本内容
|
|
||||||
"""
|
|
||||||
import pdfminer
|
|
||||||
from pdfminer.pdfparser import PDFParser
|
|
||||||
from pdfminer.pdfdocument import PDFDocument
|
|
||||||
from pdfminer.pdfpage import PDFPage, PDFTextExtractionNotAllowed
|
|
||||||
from pdfminer.pdfinterp import PDFResourceManager, PDFPageInterpreter
|
|
||||||
from pdfminer.pdfdevice import PDFDevice
|
|
||||||
from pdfminer.layout import LAParams
|
|
||||||
from pdfminer.converter import PDFPageAggregator
|
|
||||||
|
|
||||||
fp = open(pdfPath, 'rb')
|
|
||||||
|
|
||||||
# Create a PDF parser object associated with the file object
|
|
||||||
parser = PDFParser(fp)
|
|
||||||
|
|
||||||
# Create a PDF document object that stores the document structure.
|
|
||||||
# Password for initialization as 2nd parameter
|
|
||||||
document = PDFDocument(parser)
|
|
||||||
# Check if the document allows text extraction. If not, abort.
|
|
||||||
if not document.is_extractable:
|
|
||||||
raise PDFTextExtractionNotAllowed
|
|
||||||
|
|
||||||
# Create a PDF resource manager object that stores shared resources.
|
|
||||||
rsrcmgr = PDFResourceManager()
|
|
||||||
|
|
||||||
# Create a PDF device object.
|
|
||||||
# device = PDFDevice(rsrcmgr)
|
|
||||||
|
|
||||||
# BEGIN LAYOUT ANALYSIS.
|
|
||||||
# Set parameters for analysis.
|
|
||||||
laparams = LAParams(
|
|
||||||
char_margin=10.0,
|
|
||||||
line_margin=0.2,
|
|
||||||
boxes_flow=0.2,
|
|
||||||
all_texts=False,
|
|
||||||
)
|
|
||||||
# Create a PDF page aggregator object.
|
|
||||||
device = PDFPageAggregator(rsrcmgr, laparams=laparams)
|
|
||||||
# Create a PDF interpreter object.
|
|
||||||
interpreter = PDFPageInterpreter(rsrcmgr, device)
|
|
||||||
|
|
||||||
# loop over all pages in the document
|
|
||||||
outTextList = []
|
|
||||||
for page in PDFPage.create_pages(document):
|
|
||||||
# read the page into a layout object
|
|
||||||
interpreter.process_page(page)
|
|
||||||
layout = device.get_result()
|
|
||||||
for obj in layout._objs:
|
|
||||||
if isinstance(obj, pdfminer.layout.LTTextBoxHorizontal):
|
|
||||||
outTextList.append(obj.get_text())
|
|
||||||
|
|
||||||
return outTextList
|
|
||||||
|
|
||||||
|
|
||||||
def 解析Paper(file_manifest, project_folder, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt):
|
|
||||||
import time, glob, os
|
|
||||||
from bs4 import BeautifulSoup
|
|
||||||
logger.info('begin analysis on:', file_manifest)
|
|
||||||
for index, fp in enumerate(file_manifest):
|
|
||||||
if ".tex" in fp:
|
|
||||||
with open(fp, 'r', encoding='utf-8', errors='replace') as f:
|
|
||||||
file_content = f.read()
|
|
||||||
if ".pdf" in fp.lower():
|
|
||||||
file_content = readPdf(fp)
|
|
||||||
file_content = BeautifulSoup(''.join(file_content), features="lxml").body.text.encode('gbk', 'ignore').decode('gbk')
|
|
||||||
|
|
||||||
prefix = "接下来请你逐文件分析下面的论文文件，概括其内容" if index==0 else ""
|
|
||||||
i_say = prefix + f'请对下面的文章片段用中文做一个概述，文件名是{os.path.relpath(fp, project_folder)}，文章内容是 ```{file_content}```'
|
|
||||||
i_say_show_user = prefix + f'[{index+1}/{len(file_manifest)}] 请对下面的文章片段做一个概述: {os.path.abspath(fp)}'
|
|
||||||
chatbot.append((i_say_show_user, "[Local Message] waiting gpt response."))
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
|
|
||||||
if not fast_debug:
|
|
||||||
msg = '正常'
|
|
||||||
# ** gpt request **
|
|
||||||
gpt_say = yield from request_gpt_model_in_new_thread_with_ui_alive(
|
|
||||||
inputs=i_say,
|
|
||||||
inputs_show_user=i_say_show_user,
|
|
||||||
llm_kwargs=llm_kwargs,
|
|
||||||
chatbot=chatbot,
|
|
||||||
history=[],
|
|
||||||
sys_prompt="总结文章。"
|
|
||||||
) # 带超时倒计时
|
|
||||||
chatbot[-1] = (i_say_show_user, gpt_say)
|
|
||||||
history.append(i_say_show_user); history.append(gpt_say)
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history, msg=msg) # 刷新界面
|
|
||||||
if not fast_debug: time.sleep(2)
|
|
||||||
|
|
||||||
all_file = ', '.join([os.path.relpath(fp, project_folder) for index, fp in enumerate(file_manifest)])
|
|
||||||
i_say = f'根据以上你自己的分析，对全文进行概括，用学术性语言写一段中文摘要，然后再写一段英文摘要（包括{all_file}）。'
|
|
||||||
chatbot.append((i_say, "[Local Message] waiting gpt response."))
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
|
|
||||||
if not fast_debug:
|
|
||||||
msg = '正常'
|
|
||||||
# ** gpt request **
|
|
||||||
gpt_say = yield from request_gpt_model_in_new_thread_with_ui_alive(
|
|
||||||
inputs=i_say,
|
|
||||||
inputs_show_user=i_say,
|
|
||||||
llm_kwargs=llm_kwargs,
|
|
||||||
chatbot=chatbot,
|
|
||||||
history=history,
|
|
||||||
sys_prompt="总结文章。"
|
|
||||||
) # 带超时倒计时
|
|
||||||
chatbot[-1] = (i_say, gpt_say)
|
|
||||||
history.append(i_say); history.append(gpt_say)
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history, msg=msg) # 刷新界面
|
|
||||||
res = write_history_to_file(history)
|
|
||||||
promote_file_to_downloadzone(res, chatbot=chatbot)
|
|
||||||
chatbot.append(("完成了吗？", res))
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history, msg=msg) # 刷新界面
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@CatchException
|
|
||||||
def 批量总结PDF文档pdfminer(txt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_request):
|
|
||||||
history = [] # 清空历史，以免输入溢出
|
|
||||||
import glob, os
|
|
||||||
|
|
||||||
# 基本信息：功能、贡献者
|
|
||||||
chatbot.append([
|
|
||||||
"函数插件功能？",
|
|
||||||
"批量总结PDF文档，此版本使用pdfminer插件，带token约简功能。函数插件贡献者: Euclid-Jie。"])
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
|
|
||||||
# 尝试导入依赖，如果缺少依赖，则给出安装建议
|
|
||||||
try:
|
|
||||||
import pdfminer, bs4
|
|
||||||
except:
|
|
||||||
report_exception(chatbot, history,
|
|
||||||
a = f"解析项目: {txt}",
|
|
||||||
b = f"导入软件依赖失败。使用该模块需要额外依赖，安装方法```pip install --upgrade pdfminer beautifulsoup4```。")
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
if os.path.exists(txt):
|
|
||||||
project_folder = txt
|
|
||||||
else:
|
|
||||||
if txt == "": txt = '空空如也的输入栏'
|
|
||||||
report_exception(chatbot, history, a = f"解析项目: {txt}", b = f"找不到本地项目或无权访问: {txt}")
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
file_manifest = [f for f in glob.glob(f'{project_folder}/**/*.tex', recursive=True)] + \
|
|
||||||
[f for f in glob.glob(f'{project_folder}/**/*.pdf', recursive=True)] # + \
|
|
||||||
# [f for f in glob.glob(f'{project_folder}/**/*.cpp', recursive=True)] + \
|
|
||||||
# [f for f in glob.glob(f'{project_folder}/**/*.c', recursive=True)]
|
|
||||||
if len(file_manifest) == 0:
|
|
||||||
report_exception(chatbot, history, a = f"解析项目: {txt}", b = f"找不到任何.tex或pdf文件: {txt}")
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
yield from 解析Paper(file_manifest, project_folder, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt)
|
|
||||||
|
|
||||||
@@ -1,125 +0,0 @@
|
|||||||
from toolbox import CatchException, report_exception, get_log_folder, gen_time_str
|
|
||||||
from toolbox import update_ui, promote_file_to_downloadzone, update_ui_lastest_msg, disable_auto_promotion
|
|
||||||
from toolbox import write_history_to_file, promote_file_to_downloadzone
|
|
||||||
from crazy_functions.crazy_utils import request_gpt_model_in_new_thread_with_ui_alive
|
|
||||||
from crazy_functions.crazy_utils import request_gpt_model_multi_threads_with_very_awesome_ui_and_high_efficiency
|
|
||||||
from crazy_functions.crazy_utils import read_and_clean_pdf_text
|
|
||||||
from .pdf_fns.parse_pdf import parse_pdf, get_avail_grobid_url, translate_pdf
|
|
||||||
from shared_utils.colorful import *
|
|
||||||
import copy
|
|
||||||
import os
|
|
||||||
import math
|
|
||||||
import logging
|
|
||||||
|
|
||||||
def markdown_to_dict(article_content):
|
|
||||||
import markdown
|
|
||||||
from bs4 import BeautifulSoup
|
|
||||||
cur_t = ""
|
|
||||||
cur_c = ""
|
|
||||||
results = {}
|
|
||||||
for line in article_content:
|
|
||||||
if line.startswith('#'):
|
|
||||||
if cur_t!="":
|
|
||||||
if cur_t not in results:
|
|
||||||
results.update({cur_t:cur_c.lstrip('\n')})
|
|
||||||
else:
|
|
||||||
# 处理重名的章节
|
|
||||||
results.update({cur_t + " " + gen_time_str():cur_c.lstrip('\n')})
|
|
||||||
cur_t = line.rstrip('\n')
|
|
||||||
cur_c = ""
|
|
||||||
else:
|
|
||||||
cur_c += line
|
|
||||||
results_final = {}
|
|
||||||
for k in list(results.keys()):
|
|
||||||
if k.startswith('# '):
|
|
||||||
results_final['title'] = k.split('# ')[-1]
|
|
||||||
results_final['authors'] = results.pop(k).lstrip('\n')
|
|
||||||
if k.startswith('###### Abstract'):
|
|
||||||
results_final['abstract'] = results.pop(k).lstrip('\n')
|
|
||||||
|
|
||||||
results_final_sections = []
|
|
||||||
for k,v in results.items():
|
|
||||||
results_final_sections.append({
|
|
||||||
'heading':k.lstrip("# "),
|
|
||||||
'text':v if len(v) > 0 else f"The beginning of {k.lstrip('# ')} section."
|
|
||||||
})
|
|
||||||
results_final['sections'] = results_final_sections
|
|
||||||
return results_final
|
|
||||||
|
|
||||||
|
|
||||||
@CatchException
|
|
||||||
def 批量翻译PDF文档(txt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_request):
|
|
||||||
|
|
||||||
disable_auto_promotion(chatbot)
|
|
||||||
# 基本信息：功能、贡献者
|
|
||||||
chatbot.append([
|
|
||||||
"函数插件功能？",
|
|
||||||
"批量翻译PDF文档。函数插件贡献者: Binary-Husky"])
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
|
|
||||||
# 清空历史，以免输入溢出
|
|
||||||
history = []
|
|
||||||
|
|
||||||
from crazy_functions.crazy_utils import get_files_from_everything
|
|
||||||
success, file_manifest, project_folder = get_files_from_everything(txt, type='.pdf')
|
|
||||||
if len(file_manifest) > 0:
|
|
||||||
# 尝试导入依赖，如果缺少依赖，则给出安装建议
|
|
||||||
try:
|
|
||||||
import nougat
|
|
||||||
import tiktoken
|
|
||||||
except:
|
|
||||||
report_exception(chatbot, history,
|
|
||||||
a=f"解析项目: {txt}",
|
|
||||||
b=f"导入软件依赖失败。使用该模块需要额外依赖，安装方法```pip install --upgrade nougat-ocr tiktoken```。")
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
success_mmd, file_manifest_mmd, _ = get_files_from_everything(txt, type='.mmd')
|
|
||||||
success = success or success_mmd
|
|
||||||
file_manifest += file_manifest_mmd
|
|
||||||
chatbot.append(["文件列表：", ", ".join([e.split('/')[-1] for e in file_manifest])]);
|
|
||||||
yield from update_ui( chatbot=chatbot, history=history)
|
|
||||||
# 检测输入参数，如没有给定输入参数，直接退出
|
|
||||||
if not success:
|
|
||||||
if txt == "": txt = '空空如也的输入栏'
|
|
||||||
|
|
||||||
# 如果没找到任何文件
|
|
||||||
if len(file_manifest) == 0:
|
|
||||||
report_exception(chatbot, history,
|
|
||||||
a=f"解析项目: {txt}", b=f"找不到任何.pdf拓展名的文件: {txt}")
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
|
|
||||||
# 开始正式执行任务
|
|
||||||
yield from 解析PDF_基于NOUGAT(file_manifest, project_folder, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt)
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
def 解析PDF_基于NOUGAT(file_manifest, project_folder, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt):
|
|
||||||
import copy
|
|
||||||
import tiktoken
|
|
||||||
TOKEN_LIMIT_PER_FRAGMENT = 1024
|
|
||||||
generated_conclusion_files = []
|
|
||||||
generated_html_files = []
|
|
||||||
DST_LANG = "中文"
|
|
||||||
from crazy_functions.crazy_utils import nougat_interface
|
|
||||||
from crazy_functions.pdf_fns.report_gen_html import construct_html
|
|
||||||
nougat_handle = nougat_interface()
|
|
||||||
for index, fp in enumerate(file_manifest):
|
|
||||||
if fp.endswith('pdf'):
|
|
||||||
chatbot.append(["当前进度：", f"正在解析论文，请稍候。（第一次运行时，需要花费较长时间下载NOUGAT参数）"]); yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
fpp = yield from nougat_handle.NOUGAT_parse_pdf(fp, chatbot, history)
|
|
||||||
promote_file_to_downloadzone(fpp, rename_file=os.path.basename(fpp)+'.nougat.mmd', chatbot=chatbot)
|
|
||||||
else:
|
|
||||||
chatbot.append(["当前论文无需解析：", fp]); yield from update_ui( chatbot=chatbot, history=history)
|
|
||||||
fpp = fp
|
|
||||||
with open(fpp, 'r', encoding='utf8') as f:
|
|
||||||
article_content = f.readlines()
|
|
||||||
article_dict = markdown_to_dict(article_content)
|
|
||||||
logging.info(article_dict)
|
|
||||||
yield from translate_pdf(article_dict, llm_kwargs, chatbot, fp, generated_conclusion_files, TOKEN_LIMIT_PER_FRAGMENT, DST_LANG)
|
|
||||||
|
|
||||||
chatbot.append(("给出输出文件清单", str(generated_conclusion_files + generated_html_files)))
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
|
|
||||||
|
|
||||||
@@ -1,192 +0,0 @@
|
|||||||
import os
|
|
||||||
from loguru import logger
|
|
||||||
from toolbox import CatchException, update_ui, gen_time_str, promote_file_to_downloadzone
|
|
||||||
from crazy_functions.crazy_utils import request_gpt_model_in_new_thread_with_ui_alive
|
|
||||||
from crazy_functions.crazy_utils import input_clipping
|
|
||||||
|
|
||||||
def inspect_dependency(chatbot, history):
|
|
||||||
# 尝试导入依赖，如果缺少依赖，则给出安装建议
|
|
||||||
try:
|
|
||||||
import manim
|
|
||||||
return True
|
|
||||||
except:
|
|
||||||
chatbot.append(["导入依赖失败", "使用该模块需要额外依赖，安装方法:```pip install manim manimgl```"])
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return False
|
|
||||||
|
|
||||||
def eval_manim(code):
|
|
||||||
import subprocess, sys, os, shutil
|
|
||||||
|
|
||||||
with open('gpt_log/MyAnimation.py', 'w', encoding='utf8') as f:
|
|
||||||
f.write(code)
|
|
||||||
|
|
||||||
def get_class_name(class_string):
|
|
||||||
import re
|
|
||||||
# Use regex to extract the class name
|
|
||||||
class_name = re.search(r'class (\w+)\(', class_string).group(1)
|
|
||||||
return class_name
|
|
||||||
|
|
||||||
class_name = get_class_name(code)
|
|
||||||
|
|
||||||
try:
|
|
||||||
time_str = gen_time_str()
|
|
||||||
subprocess.check_output([sys.executable, '-c', f"from gpt_log.MyAnimation import {class_name}; {class_name}().render()"])
|
|
||||||
shutil.move(f'media/videos/1080p60/{class_name}.mp4', f'gpt_log/{class_name}-{time_str}.mp4')
|
|
||||||
return f'gpt_log/{time_str}.mp4'
|
|
||||||
except subprocess.CalledProcessError as e:
|
|
||||||
output = e.output.decode()
|
|
||||||
logger.error(f"Command returned non-zero exit status {e.returncode}: {output}.")
|
|
||||||
return f"Evaluating python script failed: {e.output}."
|
|
||||||
except:
|
|
||||||
logger.error('generating mp4 failed')
|
|
||||||
return "Generating mp4 failed."
|
|
||||||
|
|
||||||
|
|
||||||
def get_code_block(reply):
|
|
||||||
import re
|
|
||||||
pattern = r"```([\s\S]*?)```" # regex pattern to match code blocks
|
|
||||||
matches = re.findall(pattern, reply) # find all code blocks in text
|
|
||||||
if len(matches) != 1:
|
|
||||||
raise RuntimeError("GPT is not generating proper code.")
|
|
||||||
return matches[0].strip('python') # code block
|
|
||||||
|
|
||||||
@CatchException
|
|
||||||
def 动画生成(txt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_request):
|
|
||||||
"""
|
|
||||||
txt 输入栏用户输入的文本，例如需要翻译的一段话，再例如一个包含了待处理文件的路径
|
|
||||||
llm_kwargs gpt模型参数，如温度和top_p等，一般原样传递下去就行
|
|
||||||
plugin_kwargs 插件模型的参数，暂时没有用武之地
|
|
||||||
chatbot 聊天显示框的句柄，用于显示给用户
|
|
||||||
history 聊天历史，前情提要
|
|
||||||
system_prompt 给gpt的静默提醒
|
|
||||||
user_request 当前用户的请求信息（IP地址等）
|
|
||||||
"""
|
|
||||||
# 清空历史，以免输入溢出
|
|
||||||
history = []
|
|
||||||
|
|
||||||
# 基本信息：功能、贡献者
|
|
||||||
chatbot.append([
|
|
||||||
"函数插件功能？",
|
|
||||||
"生成数学动画, 此插件处于开发阶段, 建议暂时不要使用, 作者: binary-husky, 插件初始化中 ..."
|
|
||||||
])
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
|
|
||||||
# 尝试导入依赖, 如果缺少依赖, 则给出安装建议
|
|
||||||
dep_ok = yield from inspect_dependency(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
if not dep_ok: return
|
|
||||||
|
|
||||||
# 输入
|
|
||||||
i_say = f'Generate a animation to show: ' + txt
|
|
||||||
demo = ["Here is some examples of manim", examples_of_manim()]
|
|
||||||
_, demo = input_clipping(inputs="", history=demo, max_token_limit=2560)
|
|
||||||
# 开始
|
|
||||||
gpt_say = yield from request_gpt_model_in_new_thread_with_ui_alive(
|
|
||||||
inputs=i_say, inputs_show_user=i_say,
|
|
||||||
llm_kwargs=llm_kwargs, chatbot=chatbot, history=demo,
|
|
||||||
sys_prompt=
|
|
||||||
r"Write a animation script with 3blue1brown's manim. "+
|
|
||||||
r"Please begin with `from manim import *`. " +
|
|
||||||
r"Answer me with a code block wrapped by ```."
|
|
||||||
)
|
|
||||||
chatbot.append(["开始生成动画", "..."])
|
|
||||||
history.extend([i_say, gpt_say])
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面 # 界面更新
|
|
||||||
|
|
||||||
# 将代码转为动画
|
|
||||||
code = get_code_block(gpt_say)
|
|
||||||
res = eval_manim(code)
|
|
||||||
|
|
||||||
chatbot.append(("生成的视频文件路径", res))
|
|
||||||
if os.path.exists(res):
|
|
||||||
promote_file_to_downloadzone(res, chatbot=chatbot)
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面 # 界面更新
|
|
||||||
|
|
||||||
# 在这里放一些网上搜集的demo，辅助gpt生成代码
|
|
||||||
def examples_of_manim():
|
|
||||||
return r"""
|
|
||||||
|
|
||||||
|
|
||||||
```
|
|
||||||
|
|
||||||
class MovingGroupToDestination(Scene):
|
|
||||||
def construct(self):
|
|
||||||
group = VGroup(Dot(LEFT), Dot(ORIGIN), Dot(RIGHT, color=RED), Dot(2 * RIGHT)).scale(1.4)
|
|
||||||
dest = Dot([4, 3, 0], color=YELLOW)
|
|
||||||
self.add(group, dest)
|
|
||||||
self.play(group.animate.shift(dest.get_center() - group[2].get_center()))
|
|
||||||
self.wait(0.5)
|
|
||||||
|
|
||||||
```
|
|
||||||
|
|
||||||
|
|
||||||
```
|
|
||||||
|
|
||||||
class LatexWithMovingFramebox(Scene):
|
|
||||||
def construct(self):
|
|
||||||
text=MathTex(
|
|
||||||
"\\frac{d}{dx}f(x)g(x)=","f(x)\\frac{d}{dx}g(x)","+",
|
|
||||||
"g(x)\\frac{d}{dx}f(x)"
|
|
||||||
)
|
|
||||||
self.play(Write(text))
|
|
||||||
framebox1 = SurroundingRectangle(text[1], buff = .1)
|
|
||||||
framebox2 = SurroundingRectangle(text[3], buff = .1)
|
|
||||||
self.play(
|
|
||||||
Create(framebox1),
|
|
||||||
)
|
|
||||||
self.wait()
|
|
||||||
self.play(
|
|
||||||
ReplacementTransform(framebox1,framebox2),
|
|
||||||
)
|
|
||||||
self.wait()
|
|
||||||
|
|
||||||
```
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
```
|
|
||||||
|
|
||||||
class PointWithTrace(Scene):
|
|
||||||
def construct(self):
|
|
||||||
path = VMobject()
|
|
||||||
dot = Dot()
|
|
||||||
path.set_points_as_corners([dot.get_center(), dot.get_center()])
|
|
||||||
def update_path(path):
|
|
||||||
previous_path = path.copy()
|
|
||||||
previous_path.add_points_as_corners([dot.get_center()])
|
|
||||||
path.become(previous_path)
|
|
||||||
path.add_updater(update_path)
|
|
||||||
self.add(path, dot)
|
|
||||||
self.play(Rotating(dot, radians=PI, about_point=RIGHT, run_time=2))
|
|
||||||
self.wait()
|
|
||||||
self.play(dot.animate.shift(UP))
|
|
||||||
self.play(dot.animate.shift(LEFT))
|
|
||||||
self.wait()
|
|
||||||
|
|
||||||
```
|
|
||||||
|
|
||||||
```
|
|
||||||
|
|
||||||
# do not use get_graph, this funciton is deprecated
|
|
||||||
|
|
||||||
class ExampleFunctionGraph(Scene):
|
|
||||||
def construct(self):
|
|
||||||
cos_func = FunctionGraph(
|
|
||||||
lambda t: np.cos(t) + 0.5 * np.cos(7 * t) + (1 / 7) * np.cos(14 * t),
|
|
||||||
color=RED,
|
|
||||||
)
|
|
||||||
|
|
||||||
sin_func_1 = FunctionGraph(
|
|
||||||
lambda t: np.sin(t) + 0.5 * np.sin(7 * t) + (1 / 7) * np.sin(14 * t),
|
|
||||||
color=BLUE,
|
|
||||||
)
|
|
||||||
|
|
||||||
sin_func_2 = FunctionGraph(
|
|
||||||
lambda t: np.sin(t) + 0.5 * np.sin(7 * t) + (1 / 7) * np.sin(14 * t),
|
|
||||||
x_range=[-4, 4],
|
|
||||||
color=GREEN,
|
|
||||||
).move_to([0, 1, 0])
|
|
||||||
|
|
||||||
self.add(cos_func, sin_func_1, sin_func_2)
|
|
||||||
|
|
||||||
```
|
|
||||||
"""
|
|
||||||
@@ -1,108 +0,0 @@
|
|||||||
from loguru import logger
|
|
||||||
from toolbox import update_ui
|
|
||||||
from toolbox import CatchException, report_exception
|
|
||||||
from crazy_functions.crazy_utils import read_and_clean_pdf_text
|
|
||||||
from crazy_functions.crazy_utils import request_gpt_model_in_new_thread_with_ui_alive
|
|
||||||
|
|
||||||
|
|
||||||
def 解析PDF(file_name, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt):
|
|
||||||
logger.info('begin analysis on:', file_name)
|
|
||||||
|
|
||||||
############################## <第 0 步，切割PDF> ##################################
|
|
||||||
# 递归地切割PDF文件，每一块（尽量是完整的一个section，比如introduction，experiment等，必要时再进行切割）
|
|
||||||
# 的长度必须小于 2500 个 Token
|
|
||||||
file_content, page_one = read_and_clean_pdf_text(file_name) # （尝试）按照章节切割PDF
|
|
||||||
file_content = file_content.encode('utf-8', 'ignore').decode() # avoid reading non-utf8 chars
|
|
||||||
page_one = str(page_one).encode('utf-8', 'ignore').decode() # avoid reading non-utf8 chars
|
|
||||||
|
|
||||||
TOKEN_LIMIT_PER_FRAGMENT = 2500
|
|
||||||
|
|
||||||
from crazy_functions.pdf_fns.breakdown_txt import breakdown_text_to_satisfy_token_limit
|
|
||||||
paper_fragments = breakdown_text_to_satisfy_token_limit(txt=file_content, limit=TOKEN_LIMIT_PER_FRAGMENT, llm_model=llm_kwargs['llm_model'])
|
|
||||||
page_one_fragments = breakdown_text_to_satisfy_token_limit(txt=str(page_one), limit=TOKEN_LIMIT_PER_FRAGMENT//4, llm_model=llm_kwargs['llm_model'])
|
|
||||||
# 为了更好的效果，我们剥离Introduction之后的部分（如果有）
|
|
||||||
paper_meta = page_one_fragments[0].split('introduction')[0].split('Introduction')[0].split('INTRODUCTION')[0]
|
|
||||||
|
|
||||||
############################## <第 1 步，从摘要中提取高价值信息，放到history中> ##################################
|
|
||||||
final_results = []
|
|
||||||
final_results.append(paper_meta)
|
|
||||||
|
|
||||||
############################## <第 2 步，迭代地历遍整个文章，提取精炼信息> ##################################
|
|
||||||
i_say_show_user = f'首先你在英文语境下通读整篇论文。'; gpt_say = "[Local Message] 收到。" # 用户提示
|
|
||||||
chatbot.append([i_say_show_user, gpt_say]); yield from update_ui(chatbot=chatbot, history=[]) # 更新UI
|
|
||||||
|
|
||||||
iteration_results = []
|
|
||||||
last_iteration_result = paper_meta # 初始值是摘要
|
|
||||||
MAX_WORD_TOTAL = 4096
|
|
||||||
n_fragment = len(paper_fragments)
|
|
||||||
if n_fragment >= 20: logger.warning('文章极长，不能达到预期效果')
|
|
||||||
for i in range(n_fragment):
|
|
||||||
NUM_OF_WORD = MAX_WORD_TOTAL // n_fragment
|
|
||||||
i_say = f"Read this section, recapitulate the content of this section with less than {NUM_OF_WORD} words: {paper_fragments[i]}"
|
|
||||||
i_say_show_user = f"[{i+1}/{n_fragment}] Read this section, recapitulate the content of this section with less than {NUM_OF_WORD} words: {paper_fragments[i][:200]} ...."
|
|
||||||
gpt_say = yield from request_gpt_model_in_new_thread_with_ui_alive(i_say, i_say_show_user, # i_say=真正给chatgpt的提问， i_say_show_user=给用户看的提问
|
|
||||||
llm_kwargs, chatbot,
|
|
||||||
history=["The main idea of the previous section is?", last_iteration_result], # 迭代上一次的结果
|
|
||||||
sys_prompt="Extract the main idea of this section, answer me with Chinese." # 提示
|
|
||||||
)
|
|
||||||
iteration_results.append(gpt_say)
|
|
||||||
last_iteration_result = gpt_say
|
|
||||||
|
|
||||||
############################## <第 3 步，整理history> ##################################
|
|
||||||
final_results.extend(iteration_results)
|
|
||||||
final_results.append(f'接下来，你是一名专业的学术教授，利用以上信息，使用中文回答我的问题。')
|
|
||||||
# 接下来两句话只显示在界面上，不起实际作用
|
|
||||||
i_say_show_user = f'接下来，你是一名专业的学术教授，利用以上信息，使用中文回答我的问题。'; gpt_say = "[Local Message] 收到。"
|
|
||||||
chatbot.append([i_say_show_user, gpt_say])
|
|
||||||
|
|
||||||
############################## <第 4 步，设置一个token上限，防止回答时Token溢出> ##################################
|
|
||||||
from crazy_functions.crazy_utils import input_clipping
|
|
||||||
_, final_results = input_clipping("", final_results, max_token_limit=3200)
|
|
||||||
yield from update_ui(chatbot=chatbot, history=final_results) # 注意这里的历史记录被替代了
|
|
||||||
|
|
||||||
|
|
||||||
@CatchException
|
|
||||||
def 理解PDF文档内容标准文件输入(txt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_request):
|
|
||||||
import glob, os
|
|
||||||
|
|
||||||
# 基本信息：功能、贡献者
|
|
||||||
chatbot.append([
|
|
||||||
"函数插件功能？",
|
|
||||||
"理解PDF论文内容，并且将结合上下文内容，进行学术解答。函数插件贡献者: Hanzoe, binary-husky"])
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
|
|
||||||
# 尝试导入依赖，如果缺少依赖，则给出安装建议
|
|
||||||
try:
|
|
||||||
import fitz
|
|
||||||
except:
|
|
||||||
report_exception(chatbot, history,
|
|
||||||
a = f"解析项目: {txt}",
|
|
||||||
b = f"导入软件依赖失败。使用该模块需要额外依赖，安装方法```pip install --upgrade pymupdf```。")
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
|
|
||||||
# 清空历史，以免输入溢出
|
|
||||||
history = []
|
|
||||||
|
|
||||||
# 检测输入参数，如没有给定输入参数，直接退出
|
|
||||||
if os.path.exists(txt):
|
|
||||||
project_folder = txt
|
|
||||||
else:
|
|
||||||
if txt == "":
|
|
||||||
txt = '空空如也的输入栏'
|
|
||||||
report_exception(chatbot, history,
|
|
||||||
a=f"解析项目: {txt}", b=f"找不到本地项目或无权访问: {txt}")
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
|
|
||||||
# 搜索需要处理的文件清单
|
|
||||||
file_manifest = [f for f in glob.glob(f'{project_folder}/**/*.pdf', recursive=True)]
|
|
||||||
# 如果没找到任何文件
|
|
||||||
if len(file_manifest) == 0:
|
|
||||||
report_exception(chatbot, history,
|
|
||||||
a=f"解析项目: {txt}", b=f"找不到任何.tex或.pdf文件: {txt}")
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
txt = file_manifest[0]
|
|
||||||
# 开始正式执行任务
|
|
||||||
yield from 解析PDF(txt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt)
|
|
||||||
@@ -1,54 +0,0 @@
|
|||||||
from loguru import logger
|
|
||||||
from toolbox import update_ui
|
|
||||||
from toolbox import CatchException, report_exception
|
|
||||||
from toolbox import write_history_to_file, promote_file_to_downloadzone
|
|
||||||
from crazy_functions.crazy_utils import request_gpt_model_in_new_thread_with_ui_alive
|
|
||||||
|
|
||||||
def 生成函数注释(file_manifest, project_folder, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt):
|
|
||||||
import time, os
|
|
||||||
logger.info('begin analysis on:', file_manifest)
|
|
||||||
for index, fp in enumerate(file_manifest):
|
|
||||||
with open(fp, 'r', encoding='utf-8', errors='replace') as f:
|
|
||||||
file_content = f.read()
|
|
||||||
|
|
||||||
i_say = f'请对下面的程序文件做一个概述，并对文件中的所有函数生成注释，使用markdown表格输出结果，文件名是{os.path.relpath(fp, project_folder)}，文件内容是 ```{file_content}```'
|
|
||||||
i_say_show_user = f'[{index+1}/{len(file_manifest)}] 请对下面的程序文件做一个概述，并对文件中的所有函数生成注释: {os.path.abspath(fp)}'
|
|
||||||
chatbot.append((i_say_show_user, "[Local Message] waiting gpt response."))
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
|
|
||||||
msg = '正常'
|
|
||||||
# ** gpt request **
|
|
||||||
gpt_say = yield from request_gpt_model_in_new_thread_with_ui_alive(
|
|
||||||
i_say, i_say_show_user, llm_kwargs, chatbot, history=[], sys_prompt=system_prompt) # 带超时倒计时
|
|
||||||
|
|
||||||
chatbot[-1] = (i_say_show_user, gpt_say)
|
|
||||||
history.append(i_say_show_user); history.append(gpt_say)
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history, msg=msg) # 刷新界面
|
|
||||||
time.sleep(2)
|
|
||||||
|
|
||||||
res = write_history_to_file(history)
|
|
||||||
promote_file_to_downloadzone(res, chatbot=chatbot)
|
|
||||||
chatbot.append(("完成了吗？", res))
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history, msg=msg) # 刷新界面
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@CatchException
|
|
||||||
def 批量生成函数注释(txt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_request):
|
|
||||||
history = [] # 清空历史，以免输入溢出
|
|
||||||
import glob, os
|
|
||||||
if os.path.exists(txt):
|
|
||||||
project_folder = txt
|
|
||||||
else:
|
|
||||||
if txt == "": txt = '空空如也的输入栏'
|
|
||||||
report_exception(chatbot, history, a = f"解析项目: {txt}", b = f"找不到本地项目或无权访问: {txt}")
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
file_manifest = [f for f in glob.glob(f'{project_folder}/**/*.py', recursive=True)] + \
|
|
||||||
[f for f in glob.glob(f'{project_folder}/**/*.cpp', recursive=True)]
|
|
||||||
|
|
||||||
if len(file_manifest) == 0:
|
|
||||||
report_exception(chatbot, history, a = f"解析项目: {txt}", b = f"找不到任何.tex文件: {txt}")
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
yield from 生成函数注释(file_manifest, project_folder, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt)
|
|
||||||
@@ -1,437 +0,0 @@
|
|||||||
from toolbox import CatchException, update_ui, report_exception
|
|
||||||
from crazy_functions.crazy_utils import request_gpt_model_in_new_thread_with_ui_alive
|
|
||||||
from crazy_functions.plugin_template.plugin_class_template import (
|
|
||||||
GptAcademicPluginTemplate,
|
|
||||||
)
|
|
||||||
from crazy_functions.plugin_template.plugin_class_template import ArgProperty
|
|
||||||
|
|
||||||
# 以下是每类图表的PROMPT
|
|
||||||
SELECT_PROMPT = """
|
|
||||||
“{subject}”
|
|
||||||
=============
|
|
||||||
以上是从文章中提取的摘要,将会使用这些摘要绘制图表。请你选择一个合适的图表类型:
|
|
||||||
1 流程图
|
|
||||||
2 序列图
|
|
||||||
3 类图
|
|
||||||
4 饼图
|
|
||||||
5 甘特图
|
|
||||||
6 状态图
|
|
||||||
7 实体关系图
|
|
||||||
8 象限提示图
|
|
||||||
不需要解释原因，仅需要输出单个不带任何标点符号的数字。
|
|
||||||
"""
|
|
||||||
# 没有思维导图!!!测试发现模型始终会优先选择思维导图
|
|
||||||
# 流程图
|
|
||||||
PROMPT_1 = """
|
|
||||||
请你给出围绕“{subject}”的逻辑关系图，使用mermaid语法，注意需要使用双引号将内容括起来。
|
|
||||||
mermaid语法举例：
|
|
||||||
```mermaid
|
|
||||||
graph TD
|
|
||||||
P("编程") --> L1("Python")
|
|
||||||
P("编程") --> L2("C")
|
|
||||||
P("编程") --> L3("C++")
|
|
||||||
P("编程") --> L4("Javascipt")
|
|
||||||
P("编程") --> L5("PHP")
|
|
||||||
```
|
|
||||||
"""
|
|
||||||
# 序列图
|
|
||||||
PROMPT_2 = """
|
|
||||||
请你给出围绕“{subject}”的序列图，使用mermaid语法。
|
|
||||||
mermaid语法举例：
|
|
||||||
```mermaid
|
|
||||||
sequenceDiagram
|
|
||||||
participant A as 用户
|
|
||||||
participant B as 系统
|
|
||||||
A->>B: 登录请求
|
|
||||||
B->>A: 登录成功
|
|
||||||
A->>B: 获取数据
|
|
||||||
B->>A: 返回数据
|
|
||||||
```
|
|
||||||
"""
|
|
||||||
# 类图
|
|
||||||
PROMPT_3 = """
|
|
||||||
请你给出围绕“{subject}”的类图，使用mermaid语法。
|
|
||||||
mermaid语法举例：
|
|
||||||
```mermaid
|
|
||||||
classDiagram
|
|
||||||
Class01 <|-- AveryLongClass : Cool
|
|
||||||
Class03 *-- Class04
|
|
||||||
Class05 o-- Class06
|
|
||||||
Class07 .. Class08
|
|
||||||
Class09 --> C2 : Where am i?
|
|
||||||
Class09 --* C3
|
|
||||||
Class09 --|> Class07
|
|
||||||
Class07 : equals()
|
|
||||||
Class07 : Object[] elementData
|
|
||||||
Class01 : size()
|
|
||||||
Class01 : int chimp
|
|
||||||
Class01 : int gorilla
|
|
||||||
Class08 <--> C2: Cool label
|
|
||||||
```
|
|
||||||
"""
|
|
||||||
# 饼图
|
|
||||||
PROMPT_4 = """
|
|
||||||
请你给出围绕“{subject}”的饼图，使用mermaid语法，注意需要使用双引号将内容括起来。
|
|
||||||
mermaid语法举例：
|
|
||||||
```mermaid
|
|
||||||
pie title Pets adopted by volunteers
|
|
||||||
"狗" : 386
|
|
||||||
"猫" : 85
|
|
||||||
"兔子" : 15
|
|
||||||
```
|
|
||||||
"""
|
|
||||||
# 甘特图
|
|
||||||
PROMPT_5 = """
|
|
||||||
请你给出围绕“{subject}”的甘特图，使用mermaid语法，注意需要使用双引号将内容括起来。
|
|
||||||
mermaid语法举例：
|
|
||||||
```mermaid
|
|
||||||
gantt
|
|
||||||
title "项目开发流程"
|
|
||||||
dateFormat YYYY-MM-DD
|
|
||||||
section "设计"
|
|
||||||
"需求分析" :done, des1, 2024-01-06,2024-01-08
|
|
||||||
"原型设计" :active, des2, 2024-01-09, 3d
|
|
||||||
"UI设计" : des3, after des2, 5d
|
|
||||||
section "开发"
|
|
||||||
"前端开发" :2024-01-20, 10d
|
|
||||||
"后端开发" :2024-01-20, 10d
|
|
||||||
```
|
|
||||||
"""
|
|
||||||
# 状态图
|
|
||||||
PROMPT_6 = """
|
|
||||||
请你给出围绕“{subject}”的状态图，使用mermaid语法，注意需要使用双引号将内容括起来。
|
|
||||||
mermaid语法举例：
|
|
||||||
```mermaid
|
|
||||||
stateDiagram-v2
|
|
||||||
[*] --> "Still"
|
|
||||||
"Still" --> [*]
|
|
||||||
"Still" --> "Moving"
|
|
||||||
"Moving" --> "Still"
|
|
||||||
"Moving" --> "Crash"
|
|
||||||
"Crash" --> [*]
|
|
||||||
```
|
|
||||||
"""
|
|
||||||
# 实体关系图
|
|
||||||
PROMPT_7 = """
|
|
||||||
请你给出围绕“{subject}”的实体关系图，使用mermaid语法。
|
|
||||||
mermaid语法举例：
|
|
||||||
```mermaid
|
|
||||||
erDiagram
|
|
||||||
CUSTOMER ||--o{ ORDER : places
|
|
||||||
ORDER ||--|{ LINE-ITEM : contains
|
|
||||||
CUSTOMER {
|
|
||||||
string name
|
|
||||||
string id
|
|
||||||
}
|
|
||||||
ORDER {
|
|
||||||
string orderNumber
|
|
||||||
date orderDate
|
|
||||||
string customerID
|
|
||||||
}
|
|
||||||
LINE-ITEM {
|
|
||||||
number quantity
|
|
||||||
string productID
|
|
||||||
}
|
|
||||||
```
|
|
||||||
"""
|
|
||||||
# 象限提示图
|
|
||||||
PROMPT_8 = """
|
|
||||||
请你给出围绕“{subject}”的象限图，使用mermaid语法，注意需要使用双引号将内容括起来。
|
|
||||||
mermaid语法举例：
|
|
||||||
```mermaid
|
|
||||||
graph LR
|
|
||||||
A["Hard skill"] --> B("Programming")
|
|
||||||
A["Hard skill"] --> C("Design")
|
|
||||||
D["Soft skill"] --> E("Coordination")
|
|
||||||
D["Soft skill"] --> F("Communication")
|
|
||||||
```
|
|
||||||
"""
|
|
||||||
# 思维导图
|
|
||||||
PROMPT_9 = """
|
|
||||||
{subject}
|
|
||||||
==========
|
|
||||||
请给出上方内容的思维导图，充分考虑其之间的逻辑，使用mermaid语法，注意需要使用双引号将内容括起来。
|
|
||||||
mermaid语法举例：
|
|
||||||
```mermaid
|
|
||||||
mindmap
|
|
||||||
root((mindmap))
|
|
||||||
("Origins")
|
|
||||||
("Long history")
|
|
||||||
::icon(fa fa-book)
|
|
||||||
("Popularisation")
|
|
||||||
("British popular psychology author Tony Buzan")
|
|
||||||
::icon(fa fa-user)
|
|
||||||
("Research")
|
|
||||||
("On effectiveness<br/>and features")
|
|
||||||
::icon(fa fa-search)
|
|
||||||
("On Automatic creation")
|
|
||||||
::icon(fa fa-robot)
|
|
||||||
("Uses")
|
|
||||||
("Creative techniques")
|
|
||||||
::icon(fa fa-lightbulb-o)
|
|
||||||
("Strategic planning")
|
|
||||||
::icon(fa fa-flag)
|
|
||||||
("Argument mapping")
|
|
||||||
::icon(fa fa-comments)
|
|
||||||
("Tools")
|
|
||||||
("Pen and paper")
|
|
||||||
::icon(fa fa-pencil)
|
|
||||||
("Mermaid")
|
|
||||||
::icon(fa fa-code)
|
|
||||||
```
|
|
||||||
"""
|
|
||||||
|
|
||||||
|
|
||||||
def 解析历史输入(history, llm_kwargs, file_manifest, chatbot, plugin_kwargs):
|
|
||||||
############################## <第 0 步，切割输入> ##################################
|
|
||||||
# 借用PDF切割中的函数对文本进行切割
|
|
||||||
TOKEN_LIMIT_PER_FRAGMENT = 2500
|
|
||||||
txt = (
|
|
||||||
str(history).encode("utf-8", "ignore").decode()
|
|
||||||
) # avoid reading non-utf8 chars
|
|
||||||
from crazy_functions.pdf_fns.breakdown_txt import (
|
|
||||||
breakdown_text_to_satisfy_token_limit,
|
|
||||||
)
|
|
||||||
|
|
||||||
txt = breakdown_text_to_satisfy_token_limit(
|
|
||||||
txt=txt, limit=TOKEN_LIMIT_PER_FRAGMENT, llm_model=llm_kwargs["llm_model"]
|
|
||||||
)
|
|
||||||
############################## <第 1 步，迭代地历遍整个文章，提取精炼信息> ##################################
|
|
||||||
results = []
|
|
||||||
MAX_WORD_TOTAL = 4096
|
|
||||||
n_txt = len(txt)
|
|
||||||
last_iteration_result = "从以下文本中提取摘要。"
|
|
||||||
|
|
||||||
for i in range(n_txt):
|
|
||||||
NUM_OF_WORD = MAX_WORD_TOTAL // n_txt
|
|
||||||
i_say = f"Read this section, recapitulate the content of this section with less than {NUM_OF_WORD} words in Chinese: {txt[i]}"
|
|
||||||
i_say_show_user = f"[{i+1}/{n_txt}] Read this section, recapitulate the content of this section with less than {NUM_OF_WORD} words: {txt[i][:200]} ...."
|
|
||||||
gpt_say = yield from request_gpt_model_in_new_thread_with_ui_alive(
|
|
||||||
i_say,
|
|
||||||
i_say_show_user, # i_say=真正给chatgpt的提问， i_say_show_user=给用户看的提问
|
|
||||||
llm_kwargs,
|
|
||||||
chatbot,
|
|
||||||
history=[
|
|
||||||
"The main content of the previous section is?",
|
|
||||||
last_iteration_result,
|
|
||||||
], # 迭代上一次的结果
|
|
||||||
sys_prompt="Extracts the main content from the text section where it is located for graphing purposes, answer me with Chinese.", # 提示
|
|
||||||
)
|
|
||||||
results.append(gpt_say)
|
|
||||||
last_iteration_result = gpt_say
|
|
||||||
############################## <第 2 步，根据整理的摘要选择图表类型> ##################################
|
|
||||||
gpt_say = str(plugin_kwargs) # 将图表类型参数赋值为插件参数
|
|
||||||
results_txt = "\n".join(results) # 合并摘要
|
|
||||||
if gpt_say not in [
|
|
||||||
"1",
|
|
||||||
"2",
|
|
||||||
"3",
|
|
||||||
"4",
|
|
||||||
"5",
|
|
||||||
"6",
|
|
||||||
"7",
|
|
||||||
"8",
|
|
||||||
"9",
|
|
||||||
]: # 如插件参数不正确则使用对话模型判断
|
|
||||||
i_say_show_user = (
|
|
||||||
f"接下来将判断适合的图表类型,如连续3次判断失败将会使用流程图进行绘制"
|
|
||||||
)
|
|
||||||
gpt_say = "[Local Message] 收到。" # 用户提示
|
|
||||||
chatbot.append([i_say_show_user, gpt_say])
|
|
||||||
yield from update_ui(chatbot=chatbot, history=[]) # 更新UI
|
|
||||||
i_say = SELECT_PROMPT.format(subject=results_txt)
|
|
||||||
i_say_show_user = f'请判断适合使用的流程图类型,其中数字对应关系为:1-流程图,2-序列图,3-类图,4-饼图,5-甘特图,6-状态图,7-实体关系图,8-象限提示图。由于不管提供文本是什么,模型大概率认为"思维导图"最合适,因此思维导图仅能通过参数调用。'
|
|
||||||
for i in range(3):
|
|
||||||
gpt_say = yield from request_gpt_model_in_new_thread_with_ui_alive(
|
|
||||||
inputs=i_say,
|
|
||||||
inputs_show_user=i_say_show_user,
|
|
||||||
llm_kwargs=llm_kwargs,
|
|
||||||
chatbot=chatbot,
|
|
||||||
history=[],
|
|
||||||
sys_prompt="",
|
|
||||||
)
|
|
||||||
if gpt_say in [
|
|
||||||
"1",
|
|
||||||
"2",
|
|
||||||
"3",
|
|
||||||
"4",
|
|
||||||
"5",
|
|
||||||
"6",
|
|
||||||
"7",
|
|
||||||
"8",
|
|
||||||
"9",
|
|
||||||
]: # 判断返回是否正确
|
|
||||||
break
|
|
||||||
if gpt_say not in ["1", "2", "3", "4", "5", "6", "7", "8", "9"]:
|
|
||||||
gpt_say = "1"
|
|
||||||
############################## <第 3 步，根据选择的图表类型绘制图表> ##################################
|
|
||||||
if gpt_say == "1":
|
|
||||||
i_say = PROMPT_1.format(subject=results_txt)
|
|
||||||
elif gpt_say == "2":
|
|
||||||
i_say = PROMPT_2.format(subject=results_txt)
|
|
||||||
elif gpt_say == "3":
|
|
||||||
i_say = PROMPT_3.format(subject=results_txt)
|
|
||||||
elif gpt_say == "4":
|
|
||||||
i_say = PROMPT_4.format(subject=results_txt)
|
|
||||||
elif gpt_say == "5":
|
|
||||||
i_say = PROMPT_5.format(subject=results_txt)
|
|
||||||
elif gpt_say == "6":
|
|
||||||
i_say = PROMPT_6.format(subject=results_txt)
|
|
||||||
elif gpt_say == "7":
|
|
||||||
i_say = PROMPT_7.replace("{subject}", results_txt) # 由于实体关系图用到了{}符号
|
|
||||||
elif gpt_say == "8":
|
|
||||||
i_say = PROMPT_8.format(subject=results_txt)
|
|
||||||
elif gpt_say == "9":
|
|
||||||
i_say = PROMPT_9.format(subject=results_txt)
|
|
||||||
i_say_show_user = f"请根据判断结果绘制相应的图表。如需绘制思维导图请使用参数调用,同时过大的图表可能需要复制到在线编辑器中进行渲染。"
|
|
||||||
gpt_say = yield from request_gpt_model_in_new_thread_with_ui_alive(
|
|
||||||
inputs=i_say,
|
|
||||||
inputs_show_user=i_say_show_user,
|
|
||||||
llm_kwargs=llm_kwargs,
|
|
||||||
chatbot=chatbot,
|
|
||||||
history=[],
|
|
||||||
sys_prompt="",
|
|
||||||
)
|
|
||||||
history.append(gpt_say)
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面 # 界面更新
|
|
||||||
|
|
||||||
|
|
||||||
@CatchException
|
|
||||||
def 生成多种Mermaid图表(
|
|
||||||
txt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, web_port
|
|
||||||
):
|
|
||||||
"""
|
|
||||||
txt 输入栏用户输入的文本，例如需要翻译的一段话，再例如一个包含了待处理文件的路径
|
|
||||||
llm_kwargs gpt模型参数，如温度和top_p等，一般原样传递下去就行
|
|
||||||
plugin_kwargs 插件模型的参数，用于灵活调整复杂功能的各种参数
|
|
||||||
chatbot 聊天显示框的句柄，用于显示给用户
|
|
||||||
history 聊天历史，前情提要
|
|
||||||
system_prompt 给gpt的静默提醒
|
|
||||||
web_port 当前软件运行的端口号
|
|
||||||
"""
|
|
||||||
import os
|
|
||||||
|
|
||||||
# 基本信息：功能、贡献者
|
|
||||||
chatbot.append(
|
|
||||||
[
|
|
||||||
"函数插件功能？",
|
|
||||||
"根据当前聊天历史或指定的路径文件(文件内容优先)绘制多种mermaid图表，将会由对话模型首先判断适合的图表类型，随后绘制图表。\
|
|
||||||
\n您也可以使用插件参数指定绘制的图表类型,函数插件贡献者: Menghuan1918",
|
|
||||||
]
|
|
||||||
)
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
|
|
||||||
if os.path.exists(txt): # 如输入区无内容则直接解析历史记录
|
|
||||||
from crazy_functions.pdf_fns.parse_word import extract_text_from_files
|
|
||||||
|
|
||||||
file_exist, final_result, page_one, file_manifest, excption = (
|
|
||||||
extract_text_from_files(txt, chatbot, history)
|
|
||||||
)
|
|
||||||
else:
|
|
||||||
file_exist = False
|
|
||||||
excption = ""
|
|
||||||
file_manifest = []
|
|
||||||
|
|
||||||
if excption != "":
|
|
||||||
if excption == "word":
|
|
||||||
report_exception(
|
|
||||||
chatbot,
|
|
||||||
history,
|
|
||||||
a=f"解析项目: {txt}",
|
|
||||||
b=f"找到了.doc文件，但是该文件格式不被支持，请先转化为.docx格式。",
|
|
||||||
)
|
|
||||||
|
|
||||||
elif excption == "pdf":
|
|
||||||
report_exception(
|
|
||||||
chatbot,
|
|
||||||
history,
|
|
||||||
a=f"解析项目: {txt}",
|
|
||||||
b=f"导入软件依赖失败。使用该模块需要额外依赖，安装方法```pip install --upgrade pymupdf```。",
|
|
||||||
)
|
|
||||||
|
|
||||||
elif excption == "word_pip":
|
|
||||||
report_exception(
|
|
||||||
chatbot,
|
|
||||||
history,
|
|
||||||
a=f"解析项目: {txt}",
|
|
||||||
b=f"导入软件依赖失败。使用该模块需要额外依赖，安装方法```pip install --upgrade python-docx pywin32```。",
|
|
||||||
)
|
|
||||||
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
|
|
||||||
else:
|
|
||||||
if not file_exist:
|
|
||||||
history.append(txt) # 如输入区不是文件则将输入区内容加入历史记录
|
|
||||||
i_say_show_user = f"首先你从历史记录中提取摘要。"
|
|
||||||
gpt_say = "[Local Message] 收到。" # 用户提示
|
|
||||||
chatbot.append([i_say_show_user, gpt_say])
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 更新UI
|
|
||||||
yield from 解析历史输入(
|
|
||||||
history, llm_kwargs, file_manifest, chatbot, plugin_kwargs
|
|
||||||
)
|
|
||||||
else:
|
|
||||||
file_num = len(file_manifest)
|
|
||||||
for i in range(file_num): # 依次处理文件
|
|
||||||
i_say_show_user = f"[{i+1}/{file_num}]处理文件{file_manifest[i]}"
|
|
||||||
gpt_say = "[Local Message] 收到。" # 用户提示
|
|
||||||
chatbot.append([i_say_show_user, gpt_say])
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 更新UI
|
|
||||||
history = [] # 如输入区内容为文件则清空历史记录
|
|
||||||
history.append(final_result[i])
|
|
||||||
yield from 解析历史输入(
|
|
||||||
history, llm_kwargs, file_manifest, chatbot, plugin_kwargs
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
class Mermaid_Gen(GptAcademicPluginTemplate):
|
|
||||||
def __init__(self):
|
|
||||||
pass
|
|
||||||
|
|
||||||
def define_arg_selection_menu(self):
|
|
||||||
gui_definition = {
|
|
||||||
"Type_of_Mermaid": ArgProperty(
|
|
||||||
title="绘制的Mermaid图表类型",
|
|
||||||
options=[
|
|
||||||
"由LLM决定",
|
|
||||||
"流程图",
|
|
||||||
"序列图",
|
|
||||||
"类图",
|
|
||||||
"饼图",
|
|
||||||
"甘特图",
|
|
||||||
"状态图",
|
|
||||||
"实体关系图",
|
|
||||||
"象限提示图",
|
|
||||||
"思维导图",
|
|
||||||
],
|
|
||||||
default_value="由LLM决定",
|
|
||||||
description="选择'由LLM决定'时将由对话模型判断适合的图表类型(不包括思维导图)，选择其他类型时将直接绘制指定的图表类型。",
|
|
||||||
type="dropdown",
|
|
||||||
).model_dump_json(),
|
|
||||||
}
|
|
||||||
return gui_definition
|
|
||||||
|
|
||||||
def execute(
|
|
||||||
txt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_request
|
|
||||||
):
|
|
||||||
options = [
|
|
||||||
"由LLM决定",
|
|
||||||
"流程图",
|
|
||||||
"序列图",
|
|
||||||
"类图",
|
|
||||||
"饼图",
|
|
||||||
"甘特图",
|
|
||||||
"状态图",
|
|
||||||
"实体关系图",
|
|
||||||
"象限提示图",
|
|
||||||
"思维导图",
|
|
||||||
]
|
|
||||||
plugin_kwargs = options.index(plugin_kwargs['Type_of_Mermaid'])
|
|
||||||
yield from 生成多种Mermaid图表(
|
|
||||||
txt,
|
|
||||||
llm_kwargs,
|
|
||||||
plugin_kwargs,
|
|
||||||
chatbot,
|
|
||||||
history,
|
|
||||||
system_prompt,
|
|
||||||
user_request,
|
|
||||||
)
|
|
||||||
@@ -1,117 +0,0 @@
|
|||||||
from toolbox import CatchException, update_ui, ProxyNetworkActivate, update_ui_lastest_msg, get_log_folder, get_user
|
|
||||||
from crazy_functions.crazy_utils import request_gpt_model_in_new_thread_with_ui_alive, get_files_from_everything
|
|
||||||
from loguru import logger
|
|
||||||
install_msg ="""
|
|
||||||
|
|
||||||
1. python -m pip install torch --index-url https://download.pytorch.org/whl/cpu
|
|
||||||
|
|
||||||
2. python -m pip install transformers protobuf langchain sentence-transformers faiss-cpu nltk beautifulsoup4 bitsandbytes tabulate icetk --upgrade
|
|
||||||
|
|
||||||
3. python -m pip install unstructured[all-docs] --upgrade
|
|
||||||
|
|
||||||
4. python -c 'import nltk; nltk.download("punkt")'
|
|
||||||
"""
|
|
||||||
|
|
||||||
@CatchException
|
|
||||||
def 知识库文件注入(txt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_request):
|
|
||||||
"""
|
|
||||||
txt 输入栏用户输入的文本，例如需要翻译的一段话，再例如一个包含了待处理文件的路径
|
|
||||||
llm_kwargs gpt模型参数, 如温度和top_p等, 一般原样传递下去就行
|
|
||||||
plugin_kwargs 插件模型的参数，暂时没有用武之地
|
|
||||||
chatbot 聊天显示框的句柄，用于显示给用户
|
|
||||||
history 聊天历史，前情提要
|
|
||||||
system_prompt 给gpt的静默提醒
|
|
||||||
user_request 当前用户的请求信息（IP地址等）
|
|
||||||
"""
|
|
||||||
history = [] # 清空历史，以免输入溢出
|
|
||||||
|
|
||||||
# < --------------------读取参数--------------- >
|
|
||||||
if ("advanced_arg" in plugin_kwargs) and (plugin_kwargs["advanced_arg"] == ""): plugin_kwargs.pop("advanced_arg")
|
|
||||||
kai_id = plugin_kwargs.get("advanced_arg", 'default')
|
|
||||||
|
|
||||||
chatbot.append((f"向`{kai_id}`知识库中添加文件。", "[Local Message] 从一批文件(txt, md, tex)中读取数据构建知识库, 然后进行问答。"))
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
|
|
||||||
# resolve deps
|
|
||||||
try:
|
|
||||||
# from zh_langchain import construct_vector_store
|
|
||||||
# from langchain.embeddings.huggingface import HuggingFaceEmbeddings
|
|
||||||
from crazy_functions.vector_fns.vector_database import knowledge_archive_interface
|
|
||||||
except Exception as e:
|
|
||||||
chatbot.append(["依赖不足", f"{str(e)}\n\n导入依赖失败。请用以下命令安装" + install_msg])
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
# from crazy_functions.crazy_utils import try_install_deps
|
|
||||||
# try_install_deps(['zh_langchain==0.2.1', 'pypinyin'], reload_m=['pypinyin', 'zh_langchain'])
|
|
||||||
# yield from update_ui_lastest_msg("安装完成，您可以再次重试。", chatbot, history)
|
|
||||||
return
|
|
||||||
|
|
||||||
# < --------------------读取文件--------------- >
|
|
||||||
file_manifest = []
|
|
||||||
spl = ["txt", "doc", "docx", "email", "epub", "html", "json", "md", "msg", "pdf", "ppt", "pptx", "rtf"]
|
|
||||||
for sp in spl:
|
|
||||||
_, file_manifest_tmp, _ = get_files_from_everything(txt, type=f'.{sp}')
|
|
||||||
file_manifest += file_manifest_tmp
|
|
||||||
|
|
||||||
if len(file_manifest) == 0:
|
|
||||||
chatbot.append(["没有找到任何可读取文件", "当前支持的格式包括: txt, md, docx, pptx, pdf, json等"])
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
|
|
||||||
# < -------------------预热文本向量化模组--------------- >
|
|
||||||
chatbot.append(['<br/>'.join(file_manifest), "正在预热文本向量化模组, 如果是第一次运行, 将消耗较长时间下载中文向量化模型..."])
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
logger.info('Checking Text2vec ...')
|
|
||||||
from langchain.embeddings.huggingface import HuggingFaceEmbeddings
|
|
||||||
with ProxyNetworkActivate('Download_LLM'): # 临时地激活代理网络
|
|
||||||
HuggingFaceEmbeddings(model_name="GanymedeNil/text2vec-large-chinese")
|
|
||||||
|
|
||||||
# < -------------------构建知识库--------------- >
|
|
||||||
chatbot.append(['<br/>'.join(file_manifest), "正在构建知识库..."])
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
logger.info('Establishing knowledge archive ...')
|
|
||||||
with ProxyNetworkActivate('Download_LLM'): # 临时地激活代理网络
|
|
||||||
kai = knowledge_archive_interface()
|
|
||||||
vs_path = get_log_folder(user=get_user(chatbot), plugin_name='vec_store')
|
|
||||||
kai.feed_archive(file_manifest=file_manifest, vs_path=vs_path, id=kai_id)
|
|
||||||
kai_files = kai.get_loaded_file(vs_path=vs_path)
|
|
||||||
kai_files = '<br/>'.join(kai_files)
|
|
||||||
# chatbot.append(['知识库构建成功', "正在将知识库存储至cookie中"])
|
|
||||||
# yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
# chatbot._cookies['langchain_plugin_embedding'] = kai.get_current_archive_id()
|
|
||||||
# chatbot._cookies['lock_plugin'] = 'crazy_functions.知识库文件注入->读取知识库作答'
|
|
||||||
# chatbot.append(['完成', "“根据知识库作答”函数插件已经接管问答系统, 提问吧! 但注意, 您接下来不能再使用其他插件了，刷新页面即可以退出知识库问答模式。"])
|
|
||||||
chatbot.append(['构建完成', f"当前知识库内的有效文件：\n\n---\n\n{kai_files}\n\n---\n\n请切换至“知识库问答”插件进行知识库访问, 或者使用此插件继续上传更多文件。"])
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面 # 由于请求gpt需要一段时间，我们先及时地做一次界面更新
|
|
||||||
|
|
||||||
@CatchException
|
|
||||||
def 读取知识库作答(txt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_request=-1):
|
|
||||||
# resolve deps
|
|
||||||
try:
|
|
||||||
# from zh_langchain import construct_vector_store
|
|
||||||
# from langchain.embeddings.huggingface import HuggingFaceEmbeddings
|
|
||||||
from crazy_functions.vector_fns.vector_database import knowledge_archive_interface
|
|
||||||
except Exception as e:
|
|
||||||
chatbot.append(["依赖不足", f"{str(e)}\n\n导入依赖失败。请用以下命令安装" + install_msg])
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
# from crazy_functions.crazy_utils import try_install_deps
|
|
||||||
# try_install_deps(['zh_langchain==0.2.1', 'pypinyin'], reload_m=['pypinyin', 'zh_langchain'])
|
|
||||||
# yield from update_ui_lastest_msg("安装完成，您可以再次重试。", chatbot, history)
|
|
||||||
return
|
|
||||||
|
|
||||||
# < ------------------- --------------- >
|
|
||||||
kai = knowledge_archive_interface()
|
|
||||||
|
|
||||||
if ("advanced_arg" in plugin_kwargs) and (plugin_kwargs["advanced_arg"] == ""): plugin_kwargs.pop("advanced_arg")
|
|
||||||
kai_id = plugin_kwargs.get("advanced_arg", 'default')
|
|
||||||
vs_path = get_log_folder(user=get_user(chatbot), plugin_name='vec_store')
|
|
||||||
resp, prompt = kai.answer_with_archive_by_id(txt, kai_id, vs_path)
|
|
||||||
|
|
||||||
chatbot.append((txt, f'[知识库 {kai_id}] ' + prompt))
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面 # 由于请求gpt需要一段时间，我们先及时地做一次界面更新
|
|
||||||
gpt_say = yield from request_gpt_model_in_new_thread_with_ui_alive(
|
|
||||||
inputs=prompt, inputs_show_user=txt,
|
|
||||||
llm_kwargs=llm_kwargs, chatbot=chatbot, history=[],
|
|
||||||
sys_prompt=system_prompt
|
|
||||||
)
|
|
||||||
history.extend((prompt, gpt_say))
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面 # 由于请求gpt需要一段时间，我们先及时地做一次界面更新
|
|
||||||
@@ -1,106 +0,0 @@
|
|||||||
from toolbox import CatchException, update_ui
|
|
||||||
from crazy_functions.crazy_utils import request_gpt_model_in_new_thread_with_ui_alive, input_clipping
|
|
||||||
import requests
|
|
||||||
from bs4 import BeautifulSoup
|
|
||||||
from request_llms.bridge_all import model_info
|
|
||||||
|
|
||||||
def google(query, proxies):
|
|
||||||
query = query # 在此处替换您要搜索的关键词
|
|
||||||
url = f"https://www.google.com/search?q={query}"
|
|
||||||
headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.61 Safari/537.36'}
|
|
||||||
response = requests.get(url, headers=headers, proxies=proxies)
|
|
||||||
soup = BeautifulSoup(response.content, 'html.parser')
|
|
||||||
results = []
|
|
||||||
for g in soup.find_all('div', class_='g'):
|
|
||||||
anchors = g.find_all('a')
|
|
||||||
if anchors:
|
|
||||||
link = anchors[0]['href']
|
|
||||||
if link.startswith('/url?q='):
|
|
||||||
link = link[7:]
|
|
||||||
if not link.startswith('http'):
|
|
||||||
continue
|
|
||||||
title = g.find('h3').text
|
|
||||||
item = {'title': title, 'link': link}
|
|
||||||
results.append(item)
|
|
||||||
|
|
||||||
# for r in results:
|
|
||||||
# print(r['link'])
|
|
||||||
return results
|
|
||||||
|
|
||||||
def scrape_text(url, proxies) -> str:
|
|
||||||
"""Scrape text from a webpage
|
|
||||||
|
|
||||||
Args:
|
|
||||||
url (str): The URL to scrape text from
|
|
||||||
|
|
||||||
Returns:
|
|
||||||
str: The scraped text
|
|
||||||
"""
|
|
||||||
headers = {
|
|
||||||
'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.61 Safari/537.36',
|
|
||||||
'Content-Type': 'text/plain',
|
|
||||||
}
|
|
||||||
try:
|
|
||||||
response = requests.get(url, headers=headers, proxies=proxies, timeout=8)
|
|
||||||
if response.encoding == "ISO-8859-1": response.encoding = response.apparent_encoding
|
|
||||||
except:
|
|
||||||
return "无法连接到该网页"
|
|
||||||
soup = BeautifulSoup(response.text, "html.parser")
|
|
||||||
for script in soup(["script", "style"]):
|
|
||||||
script.extract()
|
|
||||||
text = soup.get_text()
|
|
||||||
lines = (line.strip() for line in text.splitlines())
|
|
||||||
chunks = (phrase.strip() for line in lines for phrase in line.split(" "))
|
|
||||||
text = "\n".join(chunk for chunk in chunks if chunk)
|
|
||||||
return text
|
|
||||||
|
|
||||||
@CatchException
|
|
||||||
def 连接网络回答问题(txt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_request):
|
|
||||||
"""
|
|
||||||
txt 输入栏用户输入的文本，例如需要翻译的一段话，再例如一个包含了待处理文件的路径
|
|
||||||
llm_kwargs gpt模型参数，如温度和top_p等，一般原样传递下去就行
|
|
||||||
plugin_kwargs 插件模型的参数，暂时没有用武之地
|
|
||||||
chatbot 聊天显示框的句柄，用于显示给用户
|
|
||||||
history 聊天历史，前情提要
|
|
||||||
system_prompt 给gpt的静默提醒
|
|
||||||
user_request 当前用户的请求信息（IP地址等）
|
|
||||||
"""
|
|
||||||
history = [] # 清空历史，以免输入溢出
|
|
||||||
chatbot.append((f"请结合互联网信息回答以下问题：{txt}",
|
|
||||||
"[Local Message] 请注意，您正在调用一个[函数插件]的模板，该模板可以实现ChatGPT联网信息综合。该函数面向希望实现更多有趣功能的开发者，它可以作为创建新功能函数的模板。您若希望分享新的功能模组，请不吝PR！"))
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面 # 由于请求gpt需要一段时间，我们先及时地做一次界面更新
|
|
||||||
|
|
||||||
# ------------- < 第1步：爬取搜索引擎的结果 > -------------
|
|
||||||
from toolbox import get_conf
|
|
||||||
proxies = get_conf('proxies')
|
|
||||||
urls = google(txt, proxies)
|
|
||||||
history = []
|
|
||||||
if len(urls) == 0:
|
|
||||||
chatbot.append((f"结论：{txt}",
|
|
||||||
"[Local Message] 受到google限制，无法从google获取信息！"))
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面 # 由于请求gpt需要一段时间，我们先及时地做一次界面更新
|
|
||||||
return
|
|
||||||
# ------------- < 第2步：依次访问网页 > -------------
|
|
||||||
max_search_result = 5 # 最多收纳多少个网页的结果
|
|
||||||
for index, url in enumerate(urls[:max_search_result]):
|
|
||||||
res = scrape_text(url['link'], proxies)
|
|
||||||
history.extend([f"第{index}份搜索结果：", res])
|
|
||||||
chatbot.append([f"第{index}份搜索结果：", res[:500]+"......"])
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面 # 由于请求gpt需要一段时间，我们先及时地做一次界面更新
|
|
||||||
|
|
||||||
# ------------- < 第3步：ChatGPT综合 > -------------
|
|
||||||
i_say = f"从以上搜索结果中抽取信息，然后回答问题：{txt}"
|
|
||||||
i_say, history = input_clipping( # 裁剪输入，从最长的条目开始裁剪，防止爆token
|
|
||||||
inputs=i_say,
|
|
||||||
history=history,
|
|
||||||
max_token_limit=model_info[llm_kwargs['llm_model']]['max_token']*3//4
|
|
||||||
)
|
|
||||||
gpt_say = yield from request_gpt_model_in_new_thread_with_ui_alive(
|
|
||||||
inputs=i_say, inputs_show_user=i_say,
|
|
||||||
llm_kwargs=llm_kwargs, chatbot=chatbot, history=history,
|
|
||||||
sys_prompt="请从给定的若干条搜索结果中抽取信息，对最相关的两个搜索结果进行总结，然后回答问题。"
|
|
||||||
)
|
|
||||||
chatbot[-1] = (i_say, gpt_say)
|
|
||||||
history.append(i_say);history.append(gpt_say)
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面 # 界面更新
|
|
||||||
|
|
||||||
@@ -1,106 +0,0 @@
|
|||||||
from toolbox import CatchException, update_ui
|
|
||||||
from crazy_functions.crazy_utils import request_gpt_model_in_new_thread_with_ui_alive, input_clipping
|
|
||||||
import requests
|
|
||||||
from bs4 import BeautifulSoup
|
|
||||||
from request_llms.bridge_all import model_info
|
|
||||||
|
|
||||||
|
|
||||||
def bing_search(query, proxies=None):
|
|
||||||
query = query
|
|
||||||
url = f"https://cn.bing.com/search?q={query}"
|
|
||||||
headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.61 Safari/537.36'}
|
|
||||||
response = requests.get(url, headers=headers, proxies=proxies)
|
|
||||||
soup = BeautifulSoup(response.content, 'html.parser')
|
|
||||||
results = []
|
|
||||||
for g in soup.find_all('li', class_='b_algo'):
|
|
||||||
anchors = g.find_all('a')
|
|
||||||
if anchors:
|
|
||||||
link = anchors[0]['href']
|
|
||||||
if not link.startswith('http'):
|
|
||||||
continue
|
|
||||||
title = g.find('h2').text
|
|
||||||
item = {'title': title, 'link': link}
|
|
||||||
results.append(item)
|
|
||||||
|
|
||||||
# for r in results:
|
|
||||||
# print(r['link'])
|
|
||||||
return results
|
|
||||||
|
|
||||||
|
|
||||||
def scrape_text(url, proxies) -> str:
|
|
||||||
"""Scrape text from a webpage
|
|
||||||
|
|
||||||
Args:
|
|
||||||
url (str): The URL to scrape text from
|
|
||||||
|
|
||||||
Returns:
|
|
||||||
str: The scraped text
|
|
||||||
"""
|
|
||||||
headers = {
|
|
||||||
'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.61 Safari/537.36',
|
|
||||||
'Content-Type': 'text/plain',
|
|
||||||
}
|
|
||||||
try:
|
|
||||||
response = requests.get(url, headers=headers, proxies=proxies, timeout=8)
|
|
||||||
if response.encoding == "ISO-8859-1": response.encoding = response.apparent_encoding
|
|
||||||
except:
|
|
||||||
return "无法连接到该网页"
|
|
||||||
soup = BeautifulSoup(response.text, "html.parser")
|
|
||||||
for script in soup(["script", "style"]):
|
|
||||||
script.extract()
|
|
||||||
text = soup.get_text()
|
|
||||||
lines = (line.strip() for line in text.splitlines())
|
|
||||||
chunks = (phrase.strip() for line in lines for phrase in line.split(" "))
|
|
||||||
text = "\n".join(chunk for chunk in chunks if chunk)
|
|
||||||
return text
|
|
||||||
|
|
||||||
@CatchException
|
|
||||||
def 连接bing搜索回答问题(txt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_request):
|
|
||||||
"""
|
|
||||||
txt 输入栏用户输入的文本，例如需要翻译的一段话，再例如一个包含了待处理文件的路径
|
|
||||||
llm_kwargs gpt模型参数，如温度和top_p等，一般原样传递下去就行
|
|
||||||
plugin_kwargs 插件模型的参数，暂时没有用武之地
|
|
||||||
chatbot 聊天显示框的句柄，用于显示给用户
|
|
||||||
history 聊天历史，前情提要
|
|
||||||
system_prompt 给gpt的静默提醒
|
|
||||||
user_request 当前用户的请求信息（IP地址等）
|
|
||||||
"""
|
|
||||||
history = [] # 清空历史，以免输入溢出
|
|
||||||
chatbot.append((f"请结合互联网信息回答以下问题：{txt}",
|
|
||||||
"[Local Message] 请注意，您正在调用一个[函数插件]的模板，该模板可以实现ChatGPT联网信息综合。该函数面向希望实现更多有趣功能的开发者，它可以作为创建新功能函数的模板。您若希望分享新的功能模组，请不吝PR！"))
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面 # 由于请求gpt需要一段时间，我们先及时地做一次界面更新
|
|
||||||
|
|
||||||
# ------------- < 第1步：爬取搜索引擎的结果 > -------------
|
|
||||||
from toolbox import get_conf
|
|
||||||
proxies = get_conf('proxies')
|
|
||||||
urls = bing_search(txt, proxies)
|
|
||||||
history = []
|
|
||||||
if len(urls) == 0:
|
|
||||||
chatbot.append((f"结论：{txt}",
|
|
||||||
"[Local Message] 受到bing限制，无法从bing获取信息！"))
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面 # 由于请求gpt需要一段时间，我们先及时地做一次界面更新
|
|
||||||
return
|
|
||||||
# ------------- < 第2步：依次访问网页 > -------------
|
|
||||||
max_search_result = 8 # 最多收纳多少个网页的结果
|
|
||||||
for index, url in enumerate(urls[:max_search_result]):
|
|
||||||
res = scrape_text(url['link'], proxies)
|
|
||||||
history.extend([f"第{index}份搜索结果：", res])
|
|
||||||
chatbot.append([f"第{index}份搜索结果：", res[:500]+"......"])
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面 # 由于请求gpt需要一段时间，我们先及时地做一次界面更新
|
|
||||||
|
|
||||||
# ------------- < 第3步：ChatGPT综合 > -------------
|
|
||||||
i_say = f"从以上搜索结果中抽取信息，然后回答问题：{txt}"
|
|
||||||
i_say, history = input_clipping( # 裁剪输入，从最长的条目开始裁剪，防止爆token
|
|
||||||
inputs=i_say,
|
|
||||||
history=history,
|
|
||||||
max_token_limit=model_info[llm_kwargs['llm_model']]['max_token']*3//4
|
|
||||||
)
|
|
||||||
gpt_say = yield from request_gpt_model_in_new_thread_with_ui_alive(
|
|
||||||
inputs=i_say, inputs_show_user=i_say,
|
|
||||||
llm_kwargs=llm_kwargs, chatbot=chatbot, history=history,
|
|
||||||
sys_prompt="请从给定的若干条搜索结果中抽取信息，对最相关的两个搜索结果进行总结，然后回答问题。"
|
|
||||||
)
|
|
||||||
chatbot[-1] = (i_say, gpt_say)
|
|
||||||
history.append(i_say);history.append(gpt_say)
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面 # 界面更新
|
|
||||||
|
|
||||||
@@ -1,180 +0,0 @@
|
|||||||
"""
|
|
||||||
Explanation of the Void Terminal Plugin:
|
|
||||||
|
|
||||||
Please describe in natural language what you want to do.
|
|
||||||
|
|
||||||
1. You can open the plugin's dropdown menu to explore various capabilities of this project, and then describe your needs in natural language, for example:
|
|
||||||
- "Please call the plugin to translate a PDF paper for me. I just uploaded the paper to the upload area."
|
|
||||||
- "Please use the plugin to translate a PDF paper, with the address being https://www.nature.com/articles/s41586-019-1724-z.pdf."
|
|
||||||
- "Generate an image with blooming flowers and lush green grass using the plugin."
|
|
||||||
- "Translate the README using the plugin. The GitHub URL is https://github.com/facebookresearch/co-tracker."
|
|
||||||
- "Translate an Arxiv paper for me. The Arxiv ID is 1812.10695. Remember to use the plugin and don't do it manually!"
|
|
||||||
- "I don't like the current interface color. Modify the configuration and change the theme to THEME="High-Contrast"."
|
|
||||||
- "Could you please explain the structure of the Transformer network?"
|
|
||||||
|
|
||||||
2. If you use keywords like "call the plugin xxx", "modify the configuration xxx", "please", etc., your intention can be recognized more accurately.
|
|
||||||
|
|
||||||
3. Your intention can be recognized more accurately when using powerful models like GPT4. This plugin is relatively new, so please feel free to provide feedback on GitHub.
|
|
||||||
|
|
||||||
4. Now, if you need to process a file, please upload the file (drag the file to the file upload area) or describe the path to the file.
|
|
||||||
|
|
||||||
5. If you don't need to upload a file, you can simply repeat your command again.
|
|
||||||
"""
|
|
||||||
explain_msg = """
|
|
||||||
## 虚空终端插件说明:
|
|
||||||
|
|
||||||
1. 请用**自然语言**描述您需要做什么。例如：
|
|
||||||
- 「请调用插件，为我翻译PDF论文，论文我刚刚放到上传区了」
|
|
||||||
- 「请调用插件翻译PDF论文，地址为https://openreview.net/pdf?id=rJl0r3R9KX」
|
|
||||||
- 「把Arxiv论文翻译成中文PDF，arxiv论文的ID是1812.10695，记得用插件！」
|
|
||||||
- 「生成一张图片，图中鲜花怒放，绿草如茵，用插件实现」
|
|
||||||
- 「用插件翻译README，Github网址是https://github.com/facebookresearch/co-tracker」
|
|
||||||
- 「我不喜欢当前的界面颜色，修改配置，把主题THEME更换为THEME="High-Contrast"」
|
|
||||||
- 「请调用插件，解析python源代码项目，代码我刚刚打包拖到上传区了」
|
|
||||||
- 「请问Transformer网络的结构是怎样的？」
|
|
||||||
|
|
||||||
2. 您可以打开插件下拉菜单以了解本项目的各种能力。
|
|
||||||
|
|
||||||
3. 如果您使用「调用插件xxx」、「修改配置xxx」、「请问」等关键词，您的意图可以被识别的更准确。
|
|
||||||
|
|
||||||
4. 建议使用 GPT3.5 或更强的模型，弱模型可能无法理解您的想法。该插件诞生时间不长，欢迎您前往Github反馈问题。
|
|
||||||
|
|
||||||
5. 现在，如果需要处理文件，请您上传文件（将文件拖动到文件上传区），或者描述文件所在的路径。
|
|
||||||
|
|
||||||
6. 如果不需要上传文件，现在您只需要再次重复一次您的指令即可。
|
|
||||||
"""
|
|
||||||
|
|
||||||
from pydantic import BaseModel, Field
|
|
||||||
from typing import List
|
|
||||||
from toolbox import CatchException, update_ui, is_the_upload_folder
|
|
||||||
from toolbox import update_ui_lastest_msg, disable_auto_promotion
|
|
||||||
from request_llms.bridge_all import predict_no_ui_long_connection
|
|
||||||
from crazy_functions.crazy_utils import request_gpt_model_in_new_thread_with_ui_alive
|
|
||||||
from crazy_functions.crazy_utils import input_clipping
|
|
||||||
from crazy_functions.json_fns.pydantic_io import GptJsonIO, JsonStringError
|
|
||||||
from crazy_functions.vt_fns.vt_state import VoidTerminalState
|
|
||||||
from crazy_functions.vt_fns.vt_modify_config import modify_configuration_hot
|
|
||||||
from crazy_functions.vt_fns.vt_modify_config import modify_configuration_reboot
|
|
||||||
from crazy_functions.vt_fns.vt_call_plugin import execute_plugin
|
|
||||||
|
|
||||||
class UserIntention(BaseModel):
|
|
||||||
user_prompt: str = Field(description="the content of user input", default="")
|
|
||||||
intention_type: str = Field(description="the type of user intention, choose from ['ModifyConfiguration', 'ExecutePlugin', 'Chat']", default="ExecutePlugin")
|
|
||||||
user_provide_file: bool = Field(description="whether the user provides a path to a file", default=False)
|
|
||||||
user_provide_url: bool = Field(description="whether the user provides a url", default=False)
|
|
||||||
|
|
||||||
|
|
||||||
def chat(txt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_intention):
|
|
||||||
gpt_say = yield from request_gpt_model_in_new_thread_with_ui_alive(
|
|
||||||
inputs=txt, inputs_show_user=txt,
|
|
||||||
llm_kwargs=llm_kwargs, chatbot=chatbot, history=[],
|
|
||||||
sys_prompt=system_prompt
|
|
||||||
)
|
|
||||||
chatbot[-1] = [txt, gpt_say]
|
|
||||||
history.extend([txt, gpt_say])
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
pass
|
|
||||||
|
|
||||||
|
|
||||||
explain_intention_to_user = {
|
|
||||||
'Chat': "聊天对话",
|
|
||||||
'ExecutePlugin': "调用插件",
|
|
||||||
'ModifyConfiguration': "修改配置",
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def analyze_intention_with_simple_rules(txt):
|
|
||||||
user_intention = UserIntention()
|
|
||||||
user_intention.user_prompt = txt
|
|
||||||
is_certain = False
|
|
||||||
|
|
||||||
if '请问' in txt:
|
|
||||||
is_certain = True
|
|
||||||
user_intention.intention_type = 'Chat'
|
|
||||||
|
|
||||||
if '用插件' in txt:
|
|
||||||
is_certain = True
|
|
||||||
user_intention.intention_type = 'ExecutePlugin'
|
|
||||||
|
|
||||||
if '修改配置' in txt:
|
|
||||||
is_certain = True
|
|
||||||
user_intention.intention_type = 'ModifyConfiguration'
|
|
||||||
|
|
||||||
return is_certain, user_intention
|
|
||||||
|
|
||||||
|
|
||||||
@CatchException
|
|
||||||
def 虚空终端(txt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_request):
|
|
||||||
disable_auto_promotion(chatbot=chatbot)
|
|
||||||
# 获取当前虚空终端状态
|
|
||||||
state = VoidTerminalState.get_state(chatbot)
|
|
||||||
appendix_msg = ""
|
|
||||||
|
|
||||||
# 用简单的关键词检测用户意图
|
|
||||||
is_certain, _ = analyze_intention_with_simple_rules(txt)
|
|
||||||
if is_the_upload_folder(txt):
|
|
||||||
state.set_state(chatbot=chatbot, key='has_provided_explaination', value=False)
|
|
||||||
appendix_msg = "\n\n**很好，您已经上传了文件**，现在请您描述您的需求。"
|
|
||||||
|
|
||||||
if is_certain or (state.has_provided_explaination):
|
|
||||||
# 如果意图明确，跳过提示环节
|
|
||||||
state.set_state(chatbot=chatbot, key='has_provided_explaination', value=True)
|
|
||||||
state.unlock_plugin(chatbot=chatbot)
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history)
|
|
||||||
yield from 虚空终端主路由(txt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_request)
|
|
||||||
return
|
|
||||||
else:
|
|
||||||
# 如果意图模糊，提示
|
|
||||||
state.set_state(chatbot=chatbot, key='has_provided_explaination', value=True)
|
|
||||||
state.lock_plugin(chatbot=chatbot)
|
|
||||||
chatbot.append(("虚空终端状态:", explain_msg+appendix_msg))
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history)
|
|
||||||
return
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
def 虚空终端主路由(txt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_request):
|
|
||||||
history = []
|
|
||||||
chatbot.append(("虚空终端状态: ", f"正在执行任务: {txt}"))
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
|
|
||||||
# ⭐ ⭐ ⭐ 分析用户意图
|
|
||||||
is_certain, user_intention = analyze_intention_with_simple_rules(txt)
|
|
||||||
if not is_certain:
|
|
||||||
yield from update_ui_lastest_msg(
|
|
||||||
lastmsg=f"正在执行任务: {txt}\n\n分析用户意图中", chatbot=chatbot, history=history, delay=0)
|
|
||||||
gpt_json_io = GptJsonIO(UserIntention)
|
|
||||||
rf_req = "\nchoose from ['ModifyConfiguration', 'ExecutePlugin', 'Chat']"
|
|
||||||
inputs = "Analyze the intention of the user according to following user input: \n\n" + \
|
|
||||||
">> " + (txt+rf_req).rstrip('\n').replace('\n','\n>> ') + '\n\n' + gpt_json_io.format_instructions
|
|
||||||
run_gpt_fn = lambda inputs, sys_prompt: predict_no_ui_long_connection(
|
|
||||||
inputs=inputs, llm_kwargs=llm_kwargs, history=[], sys_prompt=sys_prompt, observe_window=[])
|
|
||||||
analyze_res = run_gpt_fn(inputs, "")
|
|
||||||
try:
|
|
||||||
user_intention = gpt_json_io.generate_output_auto_repair(analyze_res, run_gpt_fn)
|
|
||||||
lastmsg=f"正在执行任务: {txt}\n\n用户意图理解: 意图={explain_intention_to_user[user_intention.intention_type]}",
|
|
||||||
except JsonStringError as e:
|
|
||||||
yield from update_ui_lastest_msg(
|
|
||||||
lastmsg=f"正在执行任务: {txt}\n\n用户意图理解: 失败 当前语言模型（{llm_kwargs['llm_model']}）不能理解您的意图", chatbot=chatbot, history=history, delay=0)
|
|
||||||
return
|
|
||||||
else:
|
|
||||||
pass
|
|
||||||
|
|
||||||
yield from update_ui_lastest_msg(
|
|
||||||
lastmsg=f"正在执行任务: {txt}\n\n用户意图理解: 意图={explain_intention_to_user[user_intention.intention_type]}",
|
|
||||||
chatbot=chatbot, history=history, delay=0)
|
|
||||||
|
|
||||||
# 用户意图: 修改本项目的配置
|
|
||||||
if user_intention.intention_type == 'ModifyConfiguration':
|
|
||||||
yield from modify_configuration_reboot(txt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_intention)
|
|
||||||
|
|
||||||
# 用户意图: 调度插件
|
|
||||||
if user_intention.intention_type == 'ExecutePlugin':
|
|
||||||
yield from execute_plugin(txt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_intention)
|
|
||||||
|
|
||||||
# 用户意图: 聊天
|
|
||||||
if user_intention.intention_type == 'Chat':
|
|
||||||
yield from chat(txt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_intention)
|
|
||||||
|
|
||||||
return
|
|
||||||
|
|
||||||
@@ -1,146 +0,0 @@
|
|||||||
from toolbox import update_ui
|
|
||||||
from toolbox import CatchException, report_exception
|
|
||||||
from toolbox import write_history_to_file, promote_file_to_downloadzone
|
|
||||||
fast_debug = True
|
|
||||||
|
|
||||||
|
|
||||||
class PaperFileGroup():
|
|
||||||
def __init__(self):
|
|
||||||
self.file_paths = []
|
|
||||||
self.file_contents = []
|
|
||||||
self.sp_file_contents = []
|
|
||||||
self.sp_file_index = []
|
|
||||||
self.sp_file_tag = []
|
|
||||||
|
|
||||||
# count_token
|
|
||||||
from request_llms.bridge_all import model_info
|
|
||||||
enc = model_info["gpt-3.5-turbo"]['tokenizer']
|
|
||||||
def get_token_num(txt): return len(enc.encode(txt, disallowed_special=()))
|
|
||||||
self.get_token_num = get_token_num
|
|
||||||
|
|
||||||
def run_file_split(self, max_token_limit=1900):
|
|
||||||
"""
|
|
||||||
将长文本分离开来
|
|
||||||
"""
|
|
||||||
for index, file_content in enumerate(self.file_contents):
|
|
||||||
if self.get_token_num(file_content) < max_token_limit:
|
|
||||||
self.sp_file_contents.append(file_content)
|
|
||||||
self.sp_file_index.append(index)
|
|
||||||
self.sp_file_tag.append(self.file_paths[index])
|
|
||||||
else:
|
|
||||||
from crazy_functions.pdf_fns.breakdown_txt import breakdown_text_to_satisfy_token_limit
|
|
||||||
segments = breakdown_text_to_satisfy_token_limit(file_content, max_token_limit)
|
|
||||||
for j, segment in enumerate(segments):
|
|
||||||
self.sp_file_contents.append(segment)
|
|
||||||
self.sp_file_index.append(index)
|
|
||||||
self.sp_file_tag.append(
|
|
||||||
self.file_paths[index] + f".part-{j}.txt")
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
def parseNotebook(filename, enable_markdown=1):
|
|
||||||
import json
|
|
||||||
|
|
||||||
CodeBlocks = []
|
|
||||||
with open(filename, 'r', encoding='utf-8', errors='replace') as f:
|
|
||||||
notebook = json.load(f)
|
|
||||||
for cell in notebook['cells']:
|
|
||||||
if cell['cell_type'] == 'code' and cell['source']:
|
|
||||||
# remove blank lines
|
|
||||||
cell['source'] = [line for line in cell['source'] if line.strip()
|
|
||||||
!= '']
|
|
||||||
CodeBlocks.append("".join(cell['source']))
|
|
||||||
elif enable_markdown and cell['cell_type'] == 'markdown' and cell['source']:
|
|
||||||
cell['source'] = [line for line in cell['source'] if line.strip()
|
|
||||||
!= '']
|
|
||||||
CodeBlocks.append("Markdown:"+"".join(cell['source']))
|
|
||||||
|
|
||||||
Code = ""
|
|
||||||
for idx, code in enumerate(CodeBlocks):
|
|
||||||
Code += f"This is {idx+1}th code block: \n"
|
|
||||||
Code += code+"\n"
|
|
||||||
|
|
||||||
return Code
|
|
||||||
|
|
||||||
|
|
||||||
def ipynb解释(file_manifest, project_folder, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt):
|
|
||||||
from crazy_functions.crazy_utils import request_gpt_model_multi_threads_with_very_awesome_ui_and_high_efficiency
|
|
||||||
|
|
||||||
if ("advanced_arg" in plugin_kwargs) and (plugin_kwargs["advanced_arg"] == ""): plugin_kwargs.pop("advanced_arg")
|
|
||||||
enable_markdown = plugin_kwargs.get("advanced_arg", "1")
|
|
||||||
try:
|
|
||||||
enable_markdown = int(enable_markdown)
|
|
||||||
except ValueError:
|
|
||||||
enable_markdown = 1
|
|
||||||
|
|
||||||
pfg = PaperFileGroup()
|
|
||||||
|
|
||||||
for fp in file_manifest:
|
|
||||||
file_content = parseNotebook(fp, enable_markdown=enable_markdown)
|
|
||||||
pfg.file_paths.append(fp)
|
|
||||||
pfg.file_contents.append(file_content)
|
|
||||||
|
|
||||||
# <-------- 拆分过长的IPynb文件 ---------->
|
|
||||||
pfg.run_file_split(max_token_limit=1024)
|
|
||||||
n_split = len(pfg.sp_file_contents)
|
|
||||||
|
|
||||||
inputs_array = [r"This is a Jupyter Notebook file, tell me about Each Block in Chinese. Focus Just On Code." +
|
|
||||||
r"If a block starts with `Markdown` which means it's a markdown block in ipynbipynb. " +
|
|
||||||
r"Start a new line for a block and block num use Chinese." +
|
|
||||||
f"\n\n{frag}" for frag in pfg.sp_file_contents]
|
|
||||||
inputs_show_user_array = [f"{f}的分析如下" for f in pfg.sp_file_tag]
|
|
||||||
sys_prompt_array = ["You are a professional programmer."] * n_split
|
|
||||||
|
|
||||||
gpt_response_collection = yield from request_gpt_model_multi_threads_with_very_awesome_ui_and_high_efficiency(
|
|
||||||
inputs_array=inputs_array,
|
|
||||||
inputs_show_user_array=inputs_show_user_array,
|
|
||||||
llm_kwargs=llm_kwargs,
|
|
||||||
chatbot=chatbot,
|
|
||||||
history_array=[[""] for _ in range(n_split)],
|
|
||||||
sys_prompt_array=sys_prompt_array,
|
|
||||||
# max_workers=5, # OpenAI所允许的最大并行过载
|
|
||||||
scroller_max_len=80
|
|
||||||
)
|
|
||||||
|
|
||||||
# <-------- 整理结果，退出 ---------->
|
|
||||||
block_result = " \n".join(gpt_response_collection)
|
|
||||||
chatbot.append(("解析的结果如下", block_result))
|
|
||||||
history.extend(["解析的结果如下", block_result])
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
|
|
||||||
# <-------- 写入文件，退出 ---------->
|
|
||||||
res = write_history_to_file(history)
|
|
||||||
promote_file_to_downloadzone(res, chatbot=chatbot)
|
|
||||||
chatbot.append(("完成了吗？", res))
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
|
|
||||||
@CatchException
|
|
||||||
def 解析ipynb文件(txt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_request):
|
|
||||||
chatbot.append([
|
|
||||||
"函数插件功能？",
|
|
||||||
"对IPynb文件进行解析。Contributor: codycjy."])
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
|
|
||||||
history = [] # 清空历史
|
|
||||||
import glob
|
|
||||||
import os
|
|
||||||
if os.path.exists(txt):
|
|
||||||
project_folder = txt
|
|
||||||
else:
|
|
||||||
if txt == "":
|
|
||||||
txt = '空空如也的输入栏'
|
|
||||||
report_exception(chatbot, history,
|
|
||||||
a=f"解析项目: {txt}", b=f"找不到本地项目或无权访问: {txt}")
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
if txt.endswith('.ipynb'):
|
|
||||||
file_manifest = [txt]
|
|
||||||
else:
|
|
||||||
file_manifest = [f for f in glob.glob(
|
|
||||||
f'{project_folder}/**/*.ipynb', recursive=True)]
|
|
||||||
if len(file_manifest) == 0:
|
|
||||||
report_exception(chatbot, history,
|
|
||||||
a=f"解析项目: {txt}", b=f"找不到任何.ipynb文件: {txt}")
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
yield from ipynb解释(file_manifest, project_folder, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, )
|
|
||||||
@@ -1,63 +0,0 @@
|
|||||||
from toolbox import CatchException, update_ui, get_conf
|
|
||||||
from crazy_functions.crazy_utils import request_gpt_model_in_new_thread_with_ui_alive
|
|
||||||
import datetime
|
|
||||||
@CatchException
|
|
||||||
def 同时问询(txt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_request):
|
|
||||||
"""
|
|
||||||
txt 输入栏用户输入的文本，例如需要翻译的一段话，再例如一个包含了待处理文件的路径
|
|
||||||
llm_kwargs gpt模型参数，如温度和top_p等，一般原样传递下去就行
|
|
||||||
plugin_kwargs 插件模型的参数，用于灵活调整复杂功能的各种参数
|
|
||||||
chatbot 聊天显示框的句柄，用于显示给用户
|
|
||||||
history 聊天历史，前情提要
|
|
||||||
system_prompt 给gpt的静默提醒
|
|
||||||
user_request 当前用户的请求信息（IP地址等）
|
|
||||||
"""
|
|
||||||
history = [] # 清空历史，以免输入溢出
|
|
||||||
MULTI_QUERY_LLM_MODELS = get_conf('MULTI_QUERY_LLM_MODELS')
|
|
||||||
chatbot.append((txt, "正在同时咨询" + MULTI_QUERY_LLM_MODELS))
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面 # 由于请求gpt需要一段时间，我们先及时地做一次界面更新
|
|
||||||
|
|
||||||
# llm_kwargs['llm_model'] = 'chatglm&gpt-3.5-turbo&api2d-gpt-3.5-turbo' # 支持任意数量的llm接口，用&符号分隔
|
|
||||||
llm_kwargs['llm_model'] = MULTI_QUERY_LLM_MODELS # 支持任意数量的llm接口，用&符号分隔
|
|
||||||
gpt_say = yield from request_gpt_model_in_new_thread_with_ui_alive(
|
|
||||||
inputs=txt, inputs_show_user=txt,
|
|
||||||
llm_kwargs=llm_kwargs, chatbot=chatbot, history=history,
|
|
||||||
sys_prompt=system_prompt,
|
|
||||||
retry_times_at_unknown_error=0
|
|
||||||
)
|
|
||||||
|
|
||||||
history.append(txt)
|
|
||||||
history.append(gpt_say)
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面 # 界面更新
|
|
||||||
|
|
||||||
|
|
||||||
@CatchException
|
|
||||||
def 同时问询_指定模型(txt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_request):
|
|
||||||
"""
|
|
||||||
txt 输入栏用户输入的文本，例如需要翻译的一段话，再例如一个包含了待处理文件的路径
|
|
||||||
llm_kwargs gpt模型参数，如温度和top_p等，一般原样传递下去就行
|
|
||||||
plugin_kwargs 插件模型的参数，用于灵活调整复杂功能的各种参数
|
|
||||||
chatbot 聊天显示框的句柄，用于显示给用户
|
|
||||||
history 聊天历史，前情提要
|
|
||||||
system_prompt 给gpt的静默提醒
|
|
||||||
user_request 当前用户的请求信息（IP地址等）
|
|
||||||
"""
|
|
||||||
history = [] # 清空历史，以免输入溢出
|
|
||||||
|
|
||||||
if ("advanced_arg" in plugin_kwargs) and (plugin_kwargs["advanced_arg"] == ""): plugin_kwargs.pop("advanced_arg")
|
|
||||||
# llm_kwargs['llm_model'] = 'chatglm&gpt-3.5-turbo&api2d-gpt-3.5-turbo' # 支持任意数量的llm接口，用&符号分隔
|
|
||||||
llm_kwargs['llm_model'] = plugin_kwargs.get("advanced_arg", 'chatglm&gpt-3.5-turbo') # 'chatglm&gpt-3.5-turbo' # 支持任意数量的llm接口，用&符号分隔
|
|
||||||
|
|
||||||
chatbot.append((txt, f"正在同时咨询{llm_kwargs['llm_model']}"))
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面 # 由于请求gpt需要一段时间，我们先及时地做一次界面更新
|
|
||||||
|
|
||||||
gpt_say = yield from request_gpt_model_in_new_thread_with_ui_alive(
|
|
||||||
inputs=txt, inputs_show_user=txt,
|
|
||||||
llm_kwargs=llm_kwargs, chatbot=chatbot, history=history,
|
|
||||||
sys_prompt=system_prompt,
|
|
||||||
retry_times_at_unknown_error=0
|
|
||||||
)
|
|
||||||
|
|
||||||
history.append(txt)
|
|
||||||
history.append(gpt_say)
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面 # 界面更新
|
|
||||||
@@ -1,194 +0,0 @@
|
|||||||
from toolbox import update_ui
|
|
||||||
from toolbox import CatchException, get_conf, markdown_convertion
|
|
||||||
from request_llms.bridge_all import predict_no_ui_long_connection
|
|
||||||
from crazy_functions.crazy_utils import input_clipping
|
|
||||||
from crazy_functions.agent_fns.watchdog import WatchDog
|
|
||||||
from crazy_functions.live_audio.aliyunASR import AliyunASR
|
|
||||||
from loguru import logger
|
|
||||||
|
|
||||||
import threading, time
|
|
||||||
import numpy as np
|
|
||||||
import json
|
|
||||||
import re
|
|
||||||
|
|
||||||
|
|
||||||
def chatbot2history(chatbot):
|
|
||||||
history = []
|
|
||||||
for c in chatbot:
|
|
||||||
for q in c:
|
|
||||||
if q in ["[ 请讲话 ]", "[ 等待GPT响应 ]", "[ 正在等您说完问题 ]"]:
|
|
||||||
continue
|
|
||||||
elif q.startswith("[ 正在等您说完问题 ]"):
|
|
||||||
continue
|
|
||||||
else:
|
|
||||||
history.append(q.strip('<div class="markdown-body">').strip('</div>').strip('<p>').strip('</p>'))
|
|
||||||
return history
|
|
||||||
|
|
||||||
def visualize_audio(chatbot, audio_shape):
|
|
||||||
if len(chatbot) == 0: chatbot.append(["[ 请讲话 ]", "[ 正在等您说完问题 ]"])
|
|
||||||
chatbot[-1] = list(chatbot[-1])
|
|
||||||
p1 = '「'
|
|
||||||
p2 = '」'
|
|
||||||
chatbot[-1][-1] = re.sub(p1+r'(.*)'+p2, '', chatbot[-1][-1])
|
|
||||||
chatbot[-1][-1] += (p1+f"`{audio_shape}`"+p2)
|
|
||||||
|
|
||||||
class AsyncGptTask():
|
|
||||||
def __init__(self) -> None:
|
|
||||||
self.observe_future = []
|
|
||||||
self.observe_future_chatbot_index = []
|
|
||||||
|
|
||||||
def gpt_thread_worker(self, i_say, llm_kwargs, history, sys_prompt, observe_window, index):
|
|
||||||
try:
|
|
||||||
MAX_TOKEN_ALLO = 2560
|
|
||||||
i_say, history = input_clipping(i_say, history, max_token_limit=MAX_TOKEN_ALLO)
|
|
||||||
gpt_say_partial = predict_no_ui_long_connection(inputs=i_say, llm_kwargs=llm_kwargs, history=history, sys_prompt=sys_prompt,
|
|
||||||
observe_window=observe_window[index], console_slience=True)
|
|
||||||
except ConnectionAbortedError as token_exceed_err:
|
|
||||||
logger.error('至少一个线程任务Token溢出而失败', e)
|
|
||||||
except Exception as e:
|
|
||||||
logger.error('至少一个线程任务意外失败', e)
|
|
||||||
|
|
||||||
def add_async_gpt_task(self, i_say, chatbot_index, llm_kwargs, history, system_prompt):
|
|
||||||
self.observe_future.append([""])
|
|
||||||
self.observe_future_chatbot_index.append(chatbot_index)
|
|
||||||
cur_index = len(self.observe_future)-1
|
|
||||||
th_new = threading.Thread(target=self.gpt_thread_worker, args=(i_say, llm_kwargs, history, system_prompt, self.observe_future, cur_index))
|
|
||||||
th_new.daemon = True
|
|
||||||
th_new.start()
|
|
||||||
|
|
||||||
def update_chatbot(self, chatbot):
|
|
||||||
for of, ofci in zip(self.observe_future, self.observe_future_chatbot_index):
|
|
||||||
try:
|
|
||||||
chatbot[ofci] = list(chatbot[ofci])
|
|
||||||
chatbot[ofci][1] = markdown_convertion(of[0])
|
|
||||||
except:
|
|
||||||
self.observe_future = []
|
|
||||||
self.observe_future_chatbot_index = []
|
|
||||||
return chatbot
|
|
||||||
|
|
||||||
class InterviewAssistant(AliyunASR):
|
|
||||||
def __init__(self):
|
|
||||||
self.capture_interval = 0.5 # second
|
|
||||||
self.stop = False
|
|
||||||
self.parsed_text = "" # 下个句子中已经说完的部分, 由 test_on_result_chg() 写入
|
|
||||||
self.parsed_sentence = "" # 某段话的整个句子, 由 test_on_sentence_end() 写入
|
|
||||||
self.buffered_sentence = "" #
|
|
||||||
self.audio_shape = "" # 音频的可视化表现, 由 audio_convertion_thread() 写入
|
|
||||||
self.event_on_result_chg = threading.Event()
|
|
||||||
self.event_on_entence_end = threading.Event()
|
|
||||||
self.event_on_commit_question = threading.Event()
|
|
||||||
|
|
||||||
def __del__(self):
|
|
||||||
self.stop = True
|
|
||||||
self.stop_msg = ""
|
|
||||||
self.commit_wd.kill_dog = True
|
|
||||||
self.plugin_wd.kill_dog = True
|
|
||||||
|
|
||||||
def init(self, chatbot):
|
|
||||||
# 初始化音频采集线程
|
|
||||||
self.captured_audio = np.array([])
|
|
||||||
self.keep_latest_n_second = 10
|
|
||||||
self.commit_after_pause_n_second = 2.0
|
|
||||||
self.ready_audio_flagment = None
|
|
||||||
self.stop = False
|
|
||||||
self.plugin_wd = WatchDog(timeout=5, bark_fn=self.__del__, msg="程序终止")
|
|
||||||
self.aut = threading.Thread(target=self.audio_convertion_thread, args=(chatbot._cookies['uuid'],))
|
|
||||||
self.aut.daemon = True
|
|
||||||
self.aut.start()
|
|
||||||
# th2 = threading.Thread(target=self.audio2txt_thread, args=(chatbot._cookies['uuid'],))
|
|
||||||
# th2.daemon = True
|
|
||||||
# th2.start()
|
|
||||||
|
|
||||||
def no_audio_for_a_while(self):
|
|
||||||
if len(self.buffered_sentence) < 7: # 如果一句话小于7个字，暂不提交
|
|
||||||
self.commit_wd.begin_watch()
|
|
||||||
else:
|
|
||||||
self.event_on_commit_question.set()
|
|
||||||
|
|
||||||
def begin(self, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt):
|
|
||||||
# main plugin function
|
|
||||||
self.init(chatbot)
|
|
||||||
chatbot.append(["[ 请讲话 ]", "[ 正在等您说完问题 ]"])
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
self.plugin_wd.begin_watch()
|
|
||||||
self.agt = AsyncGptTask()
|
|
||||||
self.commit_wd = WatchDog(timeout=self.commit_after_pause_n_second, bark_fn=self.no_audio_for_a_while, interval=0.2)
|
|
||||||
self.commit_wd.begin_watch()
|
|
||||||
|
|
||||||
while not self.stop:
|
|
||||||
self.event_on_result_chg.wait(timeout=0.25) # run once every 0.25 second
|
|
||||||
chatbot = self.agt.update_chatbot(chatbot) # 将子线程的gpt结果写入chatbot
|
|
||||||
history = chatbot2history(chatbot)
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
self.plugin_wd.feed()
|
|
||||||
|
|
||||||
if self.event_on_result_chg.is_set():
|
|
||||||
# called when some words have finished
|
|
||||||
self.event_on_result_chg.clear()
|
|
||||||
chatbot[-1] = list(chatbot[-1])
|
|
||||||
chatbot[-1][0] = self.buffered_sentence + self.parsed_text
|
|
||||||
history = chatbot2history(chatbot)
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
self.commit_wd.feed()
|
|
||||||
|
|
||||||
if self.event_on_entence_end.is_set():
|
|
||||||
# called when a sentence has ended
|
|
||||||
self.event_on_entence_end.clear()
|
|
||||||
self.parsed_text = self.parsed_sentence
|
|
||||||
self.buffered_sentence += self.parsed_text
|
|
||||||
chatbot[-1] = list(chatbot[-1])
|
|
||||||
chatbot[-1][0] = self.buffered_sentence
|
|
||||||
history = chatbot2history(chatbot)
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
|
|
||||||
if self.event_on_commit_question.is_set():
|
|
||||||
# called when a question should be commited
|
|
||||||
self.event_on_commit_question.clear()
|
|
||||||
if len(self.buffered_sentence) == 0: raise RuntimeError
|
|
||||||
|
|
||||||
self.commit_wd.begin_watch()
|
|
||||||
chatbot[-1] = list(chatbot[-1])
|
|
||||||
chatbot[-1] = [self.buffered_sentence, "[ 等待GPT响应 ]"]
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
# add gpt task 创建子线程请求gpt，避免线程阻塞
|
|
||||||
history = chatbot2history(chatbot)
|
|
||||||
self.agt.add_async_gpt_task(self.buffered_sentence, len(chatbot)-1, llm_kwargs, history, system_prompt)
|
|
||||||
|
|
||||||
self.buffered_sentence = ""
|
|
||||||
chatbot.append(["[ 请讲话 ]", "[ 正在等您说完问题 ]"])
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
|
|
||||||
if not self.event_on_result_chg.is_set() and not self.event_on_entence_end.is_set() and not self.event_on_commit_question.is_set():
|
|
||||||
visualize_audio(chatbot, self.audio_shape)
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
|
|
||||||
if len(self.stop_msg) != 0:
|
|
||||||
raise RuntimeError(self.stop_msg)
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@CatchException
|
|
||||||
def 语音助手(txt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_request):
|
|
||||||
# pip install -U openai-whisper
|
|
||||||
chatbot.append(["对话助手函数插件：使用时，双手离开鼠标键盘吧", "音频助手, 正在听您讲话（点击“停止”键可终止程序）..."])
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
|
|
||||||
# 尝试导入依赖，如果缺少依赖，则给出安装建议
|
|
||||||
try:
|
|
||||||
import nls
|
|
||||||
from scipy import io
|
|
||||||
except:
|
|
||||||
chatbot.append(["导入依赖失败", "使用该模块需要额外依赖, 安装方法:```pip install --upgrade aliyun-python-sdk-core==2.13.3 pyOpenSSL webrtcvad scipy git+https://github.com/aliyun/alibabacloud-nls-python-sdk.git```"])
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
|
|
||||||
APPKEY = get_conf('ALIYUN_APPKEY')
|
|
||||||
if APPKEY == "":
|
|
||||||
chatbot.append(["导入依赖失败", "没有阿里云语音识别APPKEY和TOKEN, 详情见https://help.aliyun.com/document_detail/450255.html"])
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
ia = InterviewAssistant()
|
|
||||||
yield from ia.begin(llm_kwargs, plugin_kwargs, chatbot, history, system_prompt)
|
|
||||||
|
|
||||||
@@ -1,63 +0,0 @@
|
|||||||
from toolbox import update_ui
|
|
||||||
from toolbox import CatchException, report_exception
|
|
||||||
from toolbox import write_history_to_file, promote_file_to_downloadzone
|
|
||||||
from crazy_functions.crazy_utils import request_gpt_model_in_new_thread_with_ui_alive
|
|
||||||
|
|
||||||
|
|
||||||
def 解析Paper(file_manifest, project_folder, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt):
|
|
||||||
import time, glob, os
|
|
||||||
for index, fp in enumerate(file_manifest):
|
|
||||||
with open(fp, 'r', encoding='utf-8', errors='replace') as f:
|
|
||||||
file_content = f.read()
|
|
||||||
|
|
||||||
prefix = "接下来请你逐文件分析下面的论文文件，概括其内容" if index==0 else ""
|
|
||||||
i_say = prefix + f'请对下面的文章片段用中文做一个概述，文件名是{os.path.relpath(fp, project_folder)}，文章内容是 ```{file_content}```'
|
|
||||||
i_say_show_user = prefix + f'[{index+1}/{len(file_manifest)}] 请对下面的文章片段做一个概述: {os.path.abspath(fp)}'
|
|
||||||
chatbot.append((i_say_show_user, "[Local Message] waiting gpt response."))
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
|
|
||||||
msg = '正常'
|
|
||||||
gpt_say = yield from request_gpt_model_in_new_thread_with_ui_alive(i_say, i_say_show_user, llm_kwargs, chatbot, history=[], sys_prompt=system_prompt) # 带超时倒计时
|
|
||||||
chatbot[-1] = (i_say_show_user, gpt_say)
|
|
||||||
history.append(i_say_show_user); history.append(gpt_say)
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history, msg=msg) # 刷新界面
|
|
||||||
time.sleep(2)
|
|
||||||
|
|
||||||
all_file = ', '.join([os.path.relpath(fp, project_folder) for index, fp in enumerate(file_manifest)])
|
|
||||||
i_say = f'根据以上你自己的分析，对全文进行概括，用学术性语言写一段中文摘要，然后再写一段英文摘要（包括{all_file}）。'
|
|
||||||
chatbot.append((i_say, "[Local Message] waiting gpt response."))
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
|
|
||||||
msg = '正常'
|
|
||||||
# ** gpt request **
|
|
||||||
gpt_say = yield from request_gpt_model_in_new_thread_with_ui_alive(i_say, i_say, llm_kwargs, chatbot, history=history, sys_prompt=system_prompt) # 带超时倒计时
|
|
||||||
|
|
||||||
chatbot[-1] = (i_say, gpt_say)
|
|
||||||
history.append(i_say); history.append(gpt_say)
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history, msg=msg) # 刷新界面
|
|
||||||
res = write_history_to_file(history)
|
|
||||||
promote_file_to_downloadzone(res, chatbot=chatbot)
|
|
||||||
chatbot.append(("完成了吗？", res))
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history, msg=msg) # 刷新界面
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@CatchException
|
|
||||||
def 读文章写摘要(txt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_request):
|
|
||||||
history = [] # 清空历史，以免输入溢出
|
|
||||||
import glob, os
|
|
||||||
if os.path.exists(txt):
|
|
||||||
project_folder = txt
|
|
||||||
else:
|
|
||||||
if txt == "": txt = '空空如也的输入栏'
|
|
||||||
report_exception(chatbot, history, a = f"解析项目: {txt}", b = f"找不到本地项目或无权访问: {txt}")
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
file_manifest = [f for f in glob.glob(f'{project_folder}/**/*.tex', recursive=True)] # + \
|
|
||||||
# [f for f in glob.glob(f'{project_folder}/**/*.cpp', recursive=True)] + \
|
|
||||||
# [f for f in glob.glob(f'{project_folder}/**/*.c', recursive=True)]
|
|
||||||
if len(file_manifest) == 0:
|
|
||||||
report_exception(chatbot, history, a = f"解析项目: {txt}", b = f"找不到任何.tex文件: {txt}")
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
yield from 解析Paper(file_manifest, project_folder, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt)
|
|
||||||
@@ -1,185 +0,0 @@
|
|||||||
from crazy_functions.crazy_utils import request_gpt_model_in_new_thread_with_ui_alive
|
|
||||||
from toolbox import CatchException, report_exception, promote_file_to_downloadzone
|
|
||||||
from toolbox import update_ui, update_ui_lastest_msg, disable_auto_promotion, write_history_to_file
|
|
||||||
import logging
|
|
||||||
import requests
|
|
||||||
import time
|
|
||||||
import random
|
|
||||||
|
|
||||||
ENABLE_ALL_VERSION_SEARCH = True
|
|
||||||
|
|
||||||
def get_meta_information(url, chatbot, history):
|
|
||||||
import arxiv
|
|
||||||
import difflib
|
|
||||||
import re
|
|
||||||
from bs4 import BeautifulSoup
|
|
||||||
from toolbox import get_conf
|
|
||||||
from urllib.parse import urlparse
|
|
||||||
session = requests.session()
|
|
||||||
|
|
||||||
proxies = get_conf('proxies')
|
|
||||||
headers = {
|
|
||||||
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36',
|
|
||||||
'Accept-Encoding': 'gzip, deflate, br',
|
|
||||||
'Accept-Language': 'en-US,en;q=0.9,zh-CN;q=0.8,zh;q=0.7',
|
|
||||||
'Cache-Control':'max-age=0',
|
|
||||||
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7',
|
|
||||||
'Connection': 'keep-alive'
|
|
||||||
}
|
|
||||||
try:
|
|
||||||
session.proxies.update(proxies)
|
|
||||||
except:
|
|
||||||
report_exception(chatbot, history,
|
|
||||||
a=f"获取代理失败 无代理状态下很可能无法访问OpenAI家族的模型及谷歌学术 建议：检查USE_PROXY选项是否修改。",
|
|
||||||
b=f"尝试直接连接")
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
session.headers.update(headers)
|
|
||||||
|
|
||||||
response = session.get(url)
|
|
||||||
# 解析网页内容
|
|
||||||
soup = BeautifulSoup(response.text, "html.parser")
|
|
||||||
|
|
||||||
def string_similar(s1, s2):
|
|
||||||
return difflib.SequenceMatcher(None, s1, s2).quick_ratio()
|
|
||||||
|
|
||||||
if ENABLE_ALL_VERSION_SEARCH:
|
|
||||||
def search_all_version(url):
|
|
||||||
time.sleep(random.randint(1,5)) # 睡一会防止触发google反爬虫
|
|
||||||
response = session.get(url)
|
|
||||||
soup = BeautifulSoup(response.text, "html.parser")
|
|
||||||
|
|
||||||
for result in soup.select(".gs_ri"):
|
|
||||||
try:
|
|
||||||
url = result.select_one(".gs_rt").a['href']
|
|
||||||
except:
|
|
||||||
continue
|
|
||||||
arxiv_id = extract_arxiv_id(url)
|
|
||||||
if not arxiv_id:
|
|
||||||
continue
|
|
||||||
search = arxiv.Search(
|
|
||||||
id_list=[arxiv_id],
|
|
||||||
max_results=1,
|
|
||||||
sort_by=arxiv.SortCriterion.Relevance,
|
|
||||||
)
|
|
||||||
try: paper = next(search.results())
|
|
||||||
except: paper = None
|
|
||||||
return paper
|
|
||||||
|
|
||||||
return None
|
|
||||||
|
|
||||||
def extract_arxiv_id(url):
|
|
||||||
# 返回给定的url解析出的arxiv_id，如url未成功匹配返回None
|
|
||||||
pattern = r'arxiv.org/abs/([^/]+)'
|
|
||||||
match = re.search(pattern, url)
|
|
||||||
if match:
|
|
||||||
return match.group(1)
|
|
||||||
else:
|
|
||||||
return None
|
|
||||||
|
|
||||||
profile = []
|
|
||||||
# 获取所有文章的标题和作者
|
|
||||||
for result in soup.select(".gs_ri"):
|
|
||||||
title = result.a.text.replace('\n', ' ').replace(' ', ' ')
|
|
||||||
author = result.select_one(".gs_a").text
|
|
||||||
try:
|
|
||||||
citation = result.select_one(".gs_fl > a[href*='cites']").text # 引用次数是链接中的文本，直接取出来
|
|
||||||
except:
|
|
||||||
citation = 'cited by 0'
|
|
||||||
abstract = result.select_one(".gs_rs").text.strip() # 摘要在 .gs_rs 中的文本，需要清除首尾空格
|
|
||||||
|
|
||||||
# 首先在arxiv上搜索，获取文章摘要
|
|
||||||
search = arxiv.Search(
|
|
||||||
query = title,
|
|
||||||
max_results = 1,
|
|
||||||
sort_by = arxiv.SortCriterion.Relevance,
|
|
||||||
)
|
|
||||||
try: paper = next(search.results())
|
|
||||||
except: paper = None
|
|
||||||
|
|
||||||
is_match = paper is not None and string_similar(title, paper.title) > 0.90
|
|
||||||
|
|
||||||
# 如果在Arxiv上匹配失败，检索文章的历史版本的题目
|
|
||||||
if not is_match and ENABLE_ALL_VERSION_SEARCH:
|
|
||||||
other_versions_page_url = [tag['href'] for tag in result.select_one('.gs_flb').select('.gs_nph') if 'cluster' in tag['href']]
|
|
||||||
if len(other_versions_page_url) > 0:
|
|
||||||
other_versions_page_url = other_versions_page_url[0]
|
|
||||||
paper = search_all_version('http://' + urlparse(url).netloc + other_versions_page_url)
|
|
||||||
is_match = paper is not None and string_similar(title, paper.title) > 0.90
|
|
||||||
|
|
||||||
if is_match:
|
|
||||||
# same paper
|
|
||||||
abstract = paper.summary.replace('\n', ' ')
|
|
||||||
is_paper_in_arxiv = True
|
|
||||||
else:
|
|
||||||
# different paper
|
|
||||||
abstract = abstract
|
|
||||||
is_paper_in_arxiv = False
|
|
||||||
|
|
||||||
logging.info('[title]:' + title)
|
|
||||||
logging.info('[author]:' + author)
|
|
||||||
logging.info('[citation]:' + citation)
|
|
||||||
|
|
||||||
profile.append({
|
|
||||||
'title': title,
|
|
||||||
'author': author,
|
|
||||||
'citation': citation,
|
|
||||||
'abstract': abstract,
|
|
||||||
'is_paper_in_arxiv': is_paper_in_arxiv,
|
|
||||||
})
|
|
||||||
|
|
||||||
chatbot[-1] = [chatbot[-1][0], title + f'\n\n是否在arxiv中（不在arxiv中无法获取完整摘要）:{is_paper_in_arxiv}\n\n' + abstract]
|
|
||||||
yield from update_ui(chatbot=chatbot, history=[]) # 刷新界面
|
|
||||||
return profile
|
|
||||||
|
|
||||||
@CatchException
|
|
||||||
def 谷歌检索小助手(txt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_request):
|
|
||||||
disable_auto_promotion(chatbot=chatbot)
|
|
||||||
# 基本信息：功能、贡献者
|
|
||||||
chatbot.append([
|
|
||||||
"函数插件功能？",
|
|
||||||
"分析用户提供的谷歌学术（google scholar）搜索页面中，出现的所有文章: binary-husky，插件初始化中..."])
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
|
|
||||||
# 尝试导入依赖，如果缺少依赖，则给出安装建议
|
|
||||||
try:
|
|
||||||
import arxiv
|
|
||||||
import math
|
|
||||||
from bs4 import BeautifulSoup
|
|
||||||
except:
|
|
||||||
report_exception(chatbot, history,
|
|
||||||
a = f"解析项目: {txt}",
|
|
||||||
b = f"导入软件依赖失败。使用该模块需要额外依赖，安装方法```pip install --upgrade beautifulsoup4 arxiv```。")
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
return
|
|
||||||
|
|
||||||
# 清空历史，以免输入溢出
|
|
||||||
history = []
|
|
||||||
meta_paper_info_list = yield from get_meta_information(txt, chatbot, history)
|
|
||||||
if len(meta_paper_info_list) == 0:
|
|
||||||
yield from update_ui_lastest_msg(lastmsg='获取文献失败，可能触发了google反爬虫机制。',chatbot=chatbot, history=history, delay=0)
|
|
||||||
return
|
|
||||||
batchsize = 5
|
|
||||||
for batch in range(math.ceil(len(meta_paper_info_list)/batchsize)):
|
|
||||||
if len(meta_paper_info_list[:batchsize]) > 0:
|
|
||||||
i_say = "下面是一些学术文献的数据，提取出以下内容：" + \
|
|
||||||
"1、英文题目；2、中文题目翻译；3、作者；4、arxiv公开（is_paper_in_arxiv）；4、引用数量（cite）；5、中文摘要翻译。" + \
|
|
||||||
f"以下是信息源：{str(meta_paper_info_list[:batchsize])}"
|
|
||||||
|
|
||||||
inputs_show_user = f"请分析此页面中出现的所有文章：{txt}，这是第{batch+1}批"
|
|
||||||
gpt_say = yield from request_gpt_model_in_new_thread_with_ui_alive(
|
|
||||||
inputs=i_say, inputs_show_user=inputs_show_user,
|
|
||||||
llm_kwargs=llm_kwargs, chatbot=chatbot, history=[],
|
|
||||||
sys_prompt="你是一个学术翻译，请从数据中提取信息。你必须使用Markdown表格。你必须逐个文献进行处理。"
|
|
||||||
)
|
|
||||||
|
|
||||||
history.extend([ f"第{batch+1}批", gpt_say ])
|
|
||||||
meta_paper_info_list = meta_paper_info_list[batchsize:]
|
|
||||||
|
|
||||||
chatbot.append(["状态？",
|
|
||||||
"已经全部完成，您可以试试让AI写一个Related Works，例如您可以继续输入Write a \"Related Works\" section about \"你搜索的研究领域\" for me."])
|
|
||||||
msg = '正常'
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history, msg=msg) # 刷新界面
|
|
||||||
path = write_history_to_file(history)
|
|
||||||
promote_file_to_downloadzone(path, chatbot=chatbot)
|
|
||||||
chatbot.append(("完成了吗？", path));
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history, msg=msg) # 刷新界面
|
|
||||||
@@ -1,54 +0,0 @@
|
|||||||
# encoding: utf-8
|
|
||||||
# @Time : 2023/4/19
|
|
||||||
# @Author : Spike
|
|
||||||
# @Descr :
|
|
||||||
from toolbox import update_ui, get_conf, get_user
|
|
||||||
from toolbox import CatchException
|
|
||||||
from toolbox import default_user_name
|
|
||||||
from crazy_functions.crazy_utils import request_gpt_model_in_new_thread_with_ui_alive
|
|
||||||
import shutil
|
|
||||||
import os
|
|
||||||
|
|
||||||
|
|
||||||
@CatchException
|
|
||||||
def 猜你想问(txt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_request):
|
|
||||||
if txt:
|
|
||||||
show_say = txt
|
|
||||||
prompt = txt+'\n回答完问题后，再列出用户可能提出的三个问题。'
|
|
||||||
else:
|
|
||||||
prompt = history[-1]+"\n分析上述回答，再列出用户可能提出的三个问题。"
|
|
||||||
show_say = '分析上述回答，再列出用户可能提出的三个问题。'
|
|
||||||
gpt_say = yield from request_gpt_model_in_new_thread_with_ui_alive(
|
|
||||||
inputs=prompt,
|
|
||||||
inputs_show_user=show_say,
|
|
||||||
llm_kwargs=llm_kwargs,
|
|
||||||
chatbot=chatbot,
|
|
||||||
history=history,
|
|
||||||
sys_prompt=system_prompt
|
|
||||||
)
|
|
||||||
chatbot[-1] = (show_say, gpt_say)
|
|
||||||
history.extend([show_say, gpt_say])
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
|
|
||||||
|
|
||||||
@CatchException
|
|
||||||
def 清除缓存(txt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_request):
|
|
||||||
chatbot.append(['清除本地缓存数据', '执行中. 删除数据'])
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
|
|
||||||
def _get_log_folder(user=default_user_name):
|
|
||||||
PATH_LOGGING = get_conf('PATH_LOGGING')
|
|
||||||
_dir = os.path.join(PATH_LOGGING, user)
|
|
||||||
if not os.path.exists(_dir): os.makedirs(_dir)
|
|
||||||
return _dir
|
|
||||||
|
|
||||||
def _get_upload_folder(user=default_user_name):
|
|
||||||
PATH_PRIVATE_UPLOAD = get_conf('PATH_PRIVATE_UPLOAD')
|
|
||||||
_dir = os.path.join(PATH_PRIVATE_UPLOAD, user)
|
|
||||||
return _dir
|
|
||||||
|
|
||||||
shutil.rmtree(_get_log_folder(get_user(chatbot)), ignore_errors=True)
|
|
||||||
shutil.rmtree(_get_upload_folder(get_user(chatbot)), ignore_errors=True)
|
|
||||||
|
|
||||||
chatbot.append(['清除本地缓存数据', '执行完成'])
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
|
|
||||||
@@ -1,153 +0,0 @@
|
|||||||
from toolbox import CatchException, update_ui
|
|
||||||
from crazy_functions.crazy_utils import request_gpt_model_in_new_thread_with_ui_alive
|
|
||||||
import datetime
|
|
||||||
|
|
||||||
####################################################################################################################
|
|
||||||
# Demo 1: 一个非常简单的插件 #########################################################################################
|
|
||||||
####################################################################################################################
|
|
||||||
|
|
||||||
高阶功能模板函数示意图 = f"""
|
|
||||||
```mermaid
|
|
||||||
flowchart TD
|
|
||||||
%% <gpt_academic_hide_mermaid_code> 一个特殊标记，用于在生成mermaid图表时隐藏代码块
|
|
||||||
subgraph 函数调用["函数调用过程"]
|
|
||||||
AA["输入栏用户输入的文本(txt)"] --> BB["gpt模型参数(llm_kwargs)"]
|
|
||||||
BB --> CC["插件模型参数(plugin_kwargs)"]
|
|
||||||
CC --> DD["对话显示框的句柄(chatbot)"]
|
|
||||||
DD --> EE["对话历史(history)"]
|
|
||||||
EE --> FF["系统提示词(system_prompt)"]
|
|
||||||
FF --> GG["当前用户信息(web_port)"]
|
|
||||||
|
|
||||||
A["开始(查询5天历史事件)"]
|
|
||||||
A --> B["获取当前月份和日期"]
|
|
||||||
B --> C["生成历史事件查询提示词"]
|
|
||||||
C --> D["调用大模型"]
|
|
||||||
D --> E["更新界面"]
|
|
||||||
E --> F["记录历史"]
|
|
||||||
F --> |"下一天"| B
|
|
||||||
end
|
|
||||||
```
|
|
||||||
"""
|
|
||||||
|
|
||||||
@CatchException
|
|
||||||
def 高阶功能模板函数(txt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_request, num_day=5):
|
|
||||||
"""
|
|
||||||
# 高阶功能模板函数示意图：https://mermaid.live/edit#pako:eNptk1tvEkEYhv8KmattQpvlvOyFCcdeeaVXuoYssBwie8gyhCIlqVoLhrbbtAWNUpEGUkyMEDW2Fmn_DDOL_8LZHdOwxrnamX3f7_3mmZk6yKhZCfAgV1KrmYKoQ9fDuKC4yChX0nld1Aou1JzjznQ5fWmejh8LYHW6vG2a47YAnlCLNSIRolnenKBXI_zRIBrcuqRT890u7jZx7zMDt-AaMbnW1--5olGiz2sQjwfoQxsZL0hxplSSU0-rop4vrzmKR6O2JxYjHmwcL2Y_HDatVMkXlf86YzHbGY9bO5j8XE7O8Nsbc3iNB3ukL2SMcH-XIQBgWoVOZzxuOxOJOyc63EPGV6ZQLENVrznViYStTiaJ2vw2M2d9bByRnOXkgCnXylCSU5quyto_IcmkbdvctELmJ-j1ASW3uB3g5xOmKqVTmqr_Na3AtuS_dtBFm8H90XJyHkDDT7S9xXWb4HGmRChx64AOL5HRpUm411rM5uh4H78Z4V7fCZzytjZz2seto9XaNPFue07clLaVZF8UNLygJ-VES8lah_n-O-5Ozc7-77NzJ0-K0yr0ZYrmHdqAk50t2RbA4qq9uNohBASw7YpSgaRkLWCCAtxAlnRZLGbJba9bPwUAC5IsCYAnn1kpJ1ZKUACC0iBSsQLVBzUlA3ioVyQ3qGhZEUrxokiehAz4nFgqk1VNVABfB1uAD_g2_AGPl-W8nMcbCvsDblADfNCz4feyobDPy3rYEMtxwYYbPFNVUoHdCPmDHBv2cP4AMfrCbiBli-Q-3afv0X6WdsIjW2-10fgDy1SAig
|
|
||||||
|
|
||||||
txt 输入栏用户输入的文本，例如需要翻译的一段话，再例如一个包含了待处理文件的路径
|
|
||||||
llm_kwargs gpt模型参数，如温度和top_p等，一般原样传递下去就行
|
|
||||||
plugin_kwargs 插件模型的参数，用于灵活调整复杂功能的各种参数
|
|
||||||
chatbot 聊天显示框的句柄，用于显示给用户
|
|
||||||
history 聊天历史，前情提要
|
|
||||||
system_prompt 给gpt的静默提醒
|
|
||||||
user_request 当前用户的请求信息（IP地址等）
|
|
||||||
"""
|
|
||||||
history = [] # 清空历史，以免输入溢出
|
|
||||||
chatbot.append((
|
|
||||||
"您正在调用插件：历史上的今天",
|
|
||||||
"[Local Message] 请注意，您正在调用一个[函数插件]的模板，该函数面向希望实现更多有趣功能的开发者，它可以作为创建新功能函数的模板（该函数只有20多行代码）。此外我们也提供可同步处理大量文件的多线程Demo供您参考。您若希望分享新的功能模组，请不吝PR！" + 高阶功能模板函数示意图))
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面 # 由于请求gpt需要一段时间，我们先及时地做一次界面更新
|
|
||||||
for i in range(int(num_day)):
|
|
||||||
currentMonth = (datetime.date.today() + datetime.timedelta(days=i)).month
|
|
||||||
currentDay = (datetime.date.today() + datetime.timedelta(days=i)).day
|
|
||||||
i_say = f'历史中哪些事件发生在{currentMonth}月{currentDay}日？列举两条并发送相关图片。发送图片时，请使用Markdown，将Unsplash API中的PUT_YOUR_QUERY_HERE替换成描述该事件的一个最重要的单词。'
|
|
||||||
gpt_say = yield from request_gpt_model_in_new_thread_with_ui_alive(
|
|
||||||
inputs=i_say, inputs_show_user=i_say,
|
|
||||||
llm_kwargs=llm_kwargs, chatbot=chatbot, history=[],
|
|
||||||
sys_prompt="当你想发送一张照片时，请使用Markdown, 并且不要有反斜线, 不要用代码块。使用 Unsplash API (https://source.unsplash.com/1280x720/? < PUT_YOUR_QUERY_HERE >)。"
|
|
||||||
)
|
|
||||||
chatbot[-1] = (i_say, gpt_say)
|
|
||||||
history.append(i_say);history.append(gpt_say)
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面 # 界面更新
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
####################################################################################################################
|
|
||||||
# Demo 2: 一个带二级菜单的插件 #######################################################################################
|
|
||||||
####################################################################################################################
|
|
||||||
|
|
||||||
from crazy_functions.plugin_template.plugin_class_template import GptAcademicPluginTemplate, ArgProperty
|
|
||||||
class Demo_Wrap(GptAcademicPluginTemplate):
|
|
||||||
def __init__(self):
|
|
||||||
"""
|
|
||||||
请注意`execute`会执行在不同的线程中，因此您在定义和使用类变量时，应当慎之又慎！
|
|
||||||
"""
|
|
||||||
pass
|
|
||||||
|
|
||||||
def define_arg_selection_menu(self):
|
|
||||||
"""
|
|
||||||
定义插件的二级选项菜单
|
|
||||||
"""
|
|
||||||
gui_definition = {
|
|
||||||
"num_day":
|
|
||||||
ArgProperty(title="日期选择", options=["仅今天", "未来3天", "未来5天"], default_value="未来3天", description="无", type="dropdown").model_dump_json(),
|
|
||||||
}
|
|
||||||
return gui_definition
|
|
||||||
|
|
||||||
def execute(txt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_request):
|
|
||||||
"""
|
|
||||||
执行插件
|
|
||||||
"""
|
|
||||||
num_day = plugin_kwargs["num_day"]
|
|
||||||
if num_day == "仅今天": num_day = 1
|
|
||||||
if num_day == "未来3天": num_day = 3
|
|
||||||
if num_day == "未来5天": num_day = 5
|
|
||||||
yield from 高阶功能模板函数(txt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_request, num_day=num_day)
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
####################################################################################################################
|
|
||||||
# Demo 3: 绘制脑图的Demo ############################################################################################
|
|
||||||
####################################################################################################################
|
|
||||||
|
|
||||||
PROMPT = """
|
|
||||||
请你给出围绕“{subject}”的逻辑关系图，使用mermaid语法，mermaid语法举例：
|
|
||||||
```mermaid
|
|
||||||
graph TD
|
|
||||||
P(编程) --> L1(Python)
|
|
||||||
P(编程) --> L2(C)
|
|
||||||
P(编程) --> L3(C++)
|
|
||||||
P(编程) --> L4(Javascipt)
|
|
||||||
P(编程) --> L5(PHP)
|
|
||||||
```
|
|
||||||
"""
|
|
||||||
@CatchException
|
|
||||||
def 测试图表渲染(txt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_request):
|
|
||||||
"""
|
|
||||||
txt 输入栏用户输入的文本，例如需要翻译的一段话，再例如一个包含了待处理文件的路径
|
|
||||||
llm_kwargs gpt模型参数，如温度和top_p等，一般原样传递下去就行
|
|
||||||
plugin_kwargs 插件模型的参数，用于灵活调整复杂功能的各种参数
|
|
||||||
chatbot 聊天显示框的句柄，用于显示给用户
|
|
||||||
history 聊天历史，前情提要
|
|
||||||
system_prompt 给gpt的静默提醒
|
|
||||||
user_request 当前用户的请求信息（IP地址等）
|
|
||||||
"""
|
|
||||||
history = [] # 清空历史，以免输入溢出
|
|
||||||
chatbot.append(("这是什么功能？", "一个测试mermaid绘制图表的功能，您可以在输入框中输入一些关键词，然后使用mermaid+llm绘制图表。"))
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面 # 由于请求gpt需要一段时间，我们先及时地做一次界面更新
|
|
||||||
|
|
||||||
if txt == "": txt = "空白的输入栏" # 调皮一下
|
|
||||||
|
|
||||||
i_say_show_user = f'请绘制有关“{txt}”的逻辑关系图。'
|
|
||||||
i_say = PROMPT.format(subject=txt)
|
|
||||||
gpt_say = yield from request_gpt_model_in_new_thread_with_ui_alive(
|
|
||||||
inputs=i_say,
|
|
||||||
inputs_show_user=i_say_show_user,
|
|
||||||
llm_kwargs=llm_kwargs, chatbot=chatbot, history=[],
|
|
||||||
sys_prompt=""
|
|
||||||
)
|
|
||||||
history.append(i_say); history.append(gpt_say)
|
|
||||||
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面 # 界面更新
|
|
||||||
@@ -1,233 +0,0 @@
|
|||||||
## ===================================================
|
|
||||||
# docker-compose.yml
|
|
||||||
## ===================================================
|
|
||||||
# 1. 请在以下方案中选择任意一种，然后删除其他的方案
|
|
||||||
# 2. 修改你选择的方案中的environment环境变量，详情请见github wiki或者config.py
|
|
||||||
# 3. 选择一种暴露服务端口的方法，并对相应的配置做出修改：
|
|
||||||
# 「方法1: 适用于Linux，很方便，可惜windows不支持」与宿主的网络融合为一体，这个是默认配置
|
|
||||||
# network_mode: "host"
|
|
||||||
# 「方法2: 适用于所有系统包括Windows和MacOS」端口映射，把容器的端口映射到宿主的端口（注意您需要先删除network_mode: "host"，再追加以下内容）
|
|
||||||
# ports:
|
|
||||||
# - "12345:12345" # 注意！12345必须与WEB_PORT环境变量相互对应
|
|
||||||
# 4. 最后`docker-compose up`运行
|
|
||||||
# 5. 如果希望使用显卡，请关注 LOCAL_MODEL_DEVICE 和 英伟达显卡运行时 选项
|
|
||||||
## ===================================================
|
|
||||||
# 1. Please choose one of the following options and delete the others.
|
|
||||||
# 2. Modify the environment variables in the selected option, see GitHub wiki or config.py for more details.
|
|
||||||
# 3. Choose a method to expose the server port and make the corresponding configuration changes:
|
|
||||||
# [Method 1: Suitable for Linux, convenient, but not supported for Windows] Fusion with the host network, this is the default configuration
|
|
||||||
# network_mode: "host"
|
|
||||||
# [Method 2: Suitable for all systems including Windows and MacOS] Port mapping, mapping the container port to the host port (note that you need to delete network_mode: "host" first, and then add the following content)
|
|
||||||
# ports:
|
|
||||||
# - "12345: 12345" # Note! 12345 must correspond to the WEB_PORT environment variable.
|
|
||||||
# 4. Finally, run `docker-compose up`.
|
|
||||||
# 5. If you want to use a graphics card, pay attention to the LOCAL_MODEL_DEVICE and Nvidia GPU runtime options.
|
|
||||||
## ===================================================
|
|
||||||
|
|
||||||
## ===================================================
|
|
||||||
## 「方案零」 部署项目的全部能力（这个是包含cuda和latex的大型镜像。如果您网速慢、硬盘小或没有显卡，则不推荐使用这个）
|
|
||||||
## ===================================================
|
|
||||||
version: '3'
|
|
||||||
services:
|
|
||||||
gpt_academic_full_capability:
|
|
||||||
image: ghcr.io/binary-husky/gpt_academic_with_all_capacity:master
|
|
||||||
environment:
|
|
||||||
# 请查阅 `config.py`或者 github wiki 以查看所有的配置信息
|
|
||||||
API_KEY: ' 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 '
|
|
||||||
# USE_PROXY: ' True '
|
|
||||||
# proxies: ' { "http": "http://localhost:10881", "https": "http://localhost:10881", } '
|
|
||||||
LLM_MODEL: ' gpt-3.5-turbo '
|
|
||||||
AVAIL_LLM_MODELS: ' ["gpt-3.5-turbo", "gpt-4", "qianfan", "sparkv2", "spark", "chatglm"] '
|
|
||||||
BAIDU_CLOUD_API_KEY : ' bTUtwEAveBrQipEowUvDwYWq '
|
|
||||||
BAIDU_CLOUD_SECRET_KEY : ' jqXtLvXiVw6UNdjliATTS61rllG8Iuni '
|
|
||||||
XFYUN_APPID: ' 53a8d816 '
|
|
||||||
XFYUN_API_SECRET: ' MjMxNDQ4NDE4MzM0OSNlNjQ2NTlhMTkx '
|
|
||||||
XFYUN_API_KEY: ' 95ccdec285364869d17b33e75ee96447 '
|
|
||||||
ENABLE_AUDIO: ' False '
|
|
||||||
DEFAULT_WORKER_NUM: ' 20 '
|
|
||||||
WEB_PORT: ' 12345 '
|
|
||||||
ADD_WAIFU: ' False '
|
|
||||||
ALIYUN_APPKEY: ' RxPlZrM88DnAFkZK '
|
|
||||||
THEME: ' Chuanhu-Small-and-Beautiful '
|
|
||||||
ALIYUN_ACCESSKEY: ' LTAI5t6BrFUzxRXVGUWnekh1 '
|
|
||||||
ALIYUN_SECRET: ' eHmI20SVWIwQZxCiTD2bGQVspP9i68 '
|
|
||||||
# LOCAL_MODEL_DEVICE: ' cuda '
|
|
||||||
|
|
||||||
# 加载英伟达显卡运行时
|
|
||||||
# runtime: nvidia
|
|
||||||
# deploy:
|
|
||||||
# resources:
|
|
||||||
# reservations:
|
|
||||||
# devices:
|
|
||||||
# - driver: nvidia
|
|
||||||
# count: 1
|
|
||||||
# capabilities: [gpu]
|
|
||||||
|
|
||||||
# 「WEB_PORT暴露方法1: 适用于Linux」与宿主的网络融合
|
|
||||||
network_mode: "host"
|
|
||||||
|
|
||||||
# 「WEB_PORT暴露方法2: 适用于所有系统」端口映射
|
|
||||||
# ports:
|
|
||||||
# - "12345:12345" # 12345必须与WEB_PORT相互对应
|
|
||||||
|
|
||||||
# 启动容器后，运行main.py主程序
|
|
||||||
command: >
|
|
||||||
bash -c "python3 -u main.py"
|
|
||||||
|
|
||||||
|
|
||||||
## ===================================================
|
|
||||||
## 「方案一」 如果不需要运行本地模型（仅 chatgpt, azure, 星火, 千帆, claude 等在线大模型服务）
|
|
||||||
## ===================================================
|
|
||||||
version: '3'
|
|
||||||
services:
|
|
||||||
gpt_academic_nolocalllms:
|
|
||||||
image: ghcr.io/binary-husky/gpt_academic_nolocal:master # (Auto Built by Dockerfile: docs/GithubAction+NoLocal)
|
|
||||||
environment:
|
|
||||||
# 请查阅 `config.py` 以查看所有的配置信息
|
|
||||||
API_KEY: ' sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx '
|
|
||||||
USE_PROXY: ' True '
|
|
||||||
proxies: ' { "http": "socks5h://localhost:10880", "https": "socks5h://localhost:10880", } '
|
|
||||||
LLM_MODEL: ' gpt-3.5-turbo '
|
|
||||||
AVAIL_LLM_MODELS: ' ["gpt-3.5-turbo", "api2d-gpt-3.5-turbo", "gpt-4", "api2d-gpt-4", "sparkv2", "qianfan"] '
|
|
||||||
WEB_PORT: ' 22303 '
|
|
||||||
ADD_WAIFU: ' True '
|
|
||||||
# THEME: ' Chuanhu-Small-and-Beautiful '
|
|
||||||
# DEFAULT_WORKER_NUM: ' 10 '
|
|
||||||
# AUTHENTICATION: ' [("username", "passwd"), ("username2", "passwd2")] '
|
|
||||||
|
|
||||||
# 「WEB_PORT暴露方法1: 适用于Linux」与宿主的网络融合
|
|
||||||
network_mode: "host"
|
|
||||||
|
|
||||||
# 启动命令
|
|
||||||
command: >
|
|
||||||
bash -c "python3 -u main.py"
|
|
||||||
|
|
||||||
|
|
||||||
### ===================================================
|
|
||||||
### 「方案二」 如果需要运行ChatGLM + Qwen + MOSS等本地模型
|
|
||||||
### ===================================================
|
|
||||||
version: '3'
|
|
||||||
services:
|
|
||||||
gpt_academic_with_chatglm:
|
|
||||||
image: ghcr.io/binary-husky/gpt_academic_chatglm_moss:master # (Auto Built by Dockerfile: docs/Dockerfile+ChatGLM)
|
|
||||||
environment:
|
|
||||||
# 请查阅 `config.py` 以查看所有的配置信息
|
|
||||||
API_KEY: ' sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx '
|
|
||||||
USE_PROXY: ' True '
|
|
||||||
proxies: ' { "http": "socks5h://localhost:10880", "https": "socks5h://localhost:10880", } '
|
|
||||||
LLM_MODEL: ' gpt-3.5-turbo '
|
|
||||||
AVAIL_LLM_MODELS: ' ["chatglm", "qwen", "moss", "gpt-3.5-turbo", "gpt-4", "newbing"] '
|
|
||||||
LOCAL_MODEL_DEVICE: ' cuda '
|
|
||||||
DEFAULT_WORKER_NUM: ' 10 '
|
|
||||||
WEB_PORT: ' 12303 '
|
|
||||||
ADD_WAIFU: ' True '
|
|
||||||
# AUTHENTICATION: ' [("username", "passwd"), ("username2", "passwd2")] '
|
|
||||||
|
|
||||||
# 显卡的使用，nvidia0指第0个GPU
|
|
||||||
runtime: nvidia
|
|
||||||
devices:
|
|
||||||
- /dev/nvidia0:/dev/nvidia0
|
|
||||||
|
|
||||||
# 「WEB_PORT暴露方法1: 适用于Linux」与宿主的网络融合
|
|
||||||
network_mode: "host"
|
|
||||||
|
|
||||||
# 启动命令
|
|
||||||
command: >
|
|
||||||
bash -c "python3 -u main.py"
|
|
||||||
|
|
||||||
# P.S. 通过对 command 进行微调，可以便捷地安装额外的依赖
|
|
||||||
# command: >
|
|
||||||
# bash -c "pip install -r request_llms/requirements_qwen.txt && python3 -u main.py"
|
|
||||||
|
|
||||||
|
|
||||||
### ===================================================
|
|
||||||
### 「方案三」 如果需要运行ChatGPT + LLAMA + 盘古 + RWKV本地模型
|
|
||||||
### ===================================================
|
|
||||||
version: '3'
|
|
||||||
services:
|
|
||||||
gpt_academic_with_rwkv:
|
|
||||||
image: ghcr.io/binary-husky/gpt_academic_jittorllms:master
|
|
||||||
environment:
|
|
||||||
# 请查阅 `config.py` 以查看所有的配置信息
|
|
||||||
API_KEY: ' sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx,fkxxxxxx-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx '
|
|
||||||
USE_PROXY: ' True '
|
|
||||||
proxies: ' { "http": "socks5h://localhost:10880", "https": "socks5h://localhost:10880", } '
|
|
||||||
LLM_MODEL: ' gpt-3.5-turbo '
|
|
||||||
AVAIL_LLM_MODELS: ' ["gpt-3.5-turbo", "newbing", "jittorllms_rwkv", "jittorllms_pangualpha", "jittorllms_llama"] '
|
|
||||||
LOCAL_MODEL_DEVICE: ' cuda '
|
|
||||||
DEFAULT_WORKER_NUM: ' 10 '
|
|
||||||
WEB_PORT: ' 12305 '
|
|
||||||
ADD_WAIFU: ' True '
|
|
||||||
# AUTHENTICATION: ' [("username", "passwd"), ("username2", "passwd2")] '
|
|
||||||
|
|
||||||
# 显卡的使用，nvidia0指第0个GPU
|
|
||||||
runtime: nvidia
|
|
||||||
devices:
|
|
||||||
- /dev/nvidia0:/dev/nvidia0
|
|
||||||
|
|
||||||
# 「WEB_PORT暴露方法1: 适用于Linux」与宿主的网络融合
|
|
||||||
network_mode: "host"
|
|
||||||
|
|
||||||
# 启动命令
|
|
||||||
command: >
|
|
||||||
python3 -u main.py
|
|
||||||
|
|
||||||
|
|
||||||
## ===================================================
|
|
||||||
## 「方案四」 ChatGPT + Latex
|
|
||||||
## ===================================================
|
|
||||||
version: '3'
|
|
||||||
services:
|
|
||||||
gpt_academic_with_latex:
|
|
||||||
image: ghcr.io/binary-husky/gpt_academic_with_latex:master # (Auto Built by Dockerfile: docs/GithubAction+NoLocal+Latex)
|
|
||||||
# 对于ARM64设备，请将以上镜像名称替换为 ghcr.io/binary-husky/gpt_academic_with_latex_arm:master
|
|
||||||
environment:
|
|
||||||
# 请查阅 `config.py` 以查看所有的配置信息
|
|
||||||
API_KEY: ' sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx '
|
|
||||||
USE_PROXY: ' True '
|
|
||||||
proxies: ' { "http": "socks5h://localhost:10880", "https": "socks5h://localhost:10880", } '
|
|
||||||
LLM_MODEL: ' gpt-3.5-turbo '
|
|
||||||
AVAIL_LLM_MODELS: ' ["gpt-3.5-turbo", "gpt-4"] '
|
|
||||||
LOCAL_MODEL_DEVICE: ' cuda '
|
|
||||||
DEFAULT_WORKER_NUM: ' 10 '
|
|
||||||
WEB_PORT: ' 12303 '
|
|
||||||
|
|
||||||
# 「WEB_PORT暴露方法1: 适用于Linux」与宿主的网络融合
|
|
||||||
network_mode: "host"
|
|
||||||
|
|
||||||
# 启动命令
|
|
||||||
command: >
|
|
||||||
bash -c "python3 -u main.py"
|
|
||||||
|
|
||||||
|
|
||||||
## ===================================================
|
|
||||||
## 「方案五」 ChatGPT + 语音助手 （请先阅读 docs/use_audio.md）
|
|
||||||
## ===================================================
|
|
||||||
version: '3'
|
|
||||||
services:
|
|
||||||
gpt_academic_with_audio:
|
|
||||||
image: ghcr.io/binary-husky/gpt_academic_audio_assistant:master
|
|
||||||
environment:
|
|
||||||
# 请查阅 `config.py` 以查看所有的配置信息
|
|
||||||
API_KEY: ' fk195831-IdP0Pb3W6DCMUIbQwVX6MsSiyxwqybyS '
|
|
||||||
USE_PROXY: ' False '
|
|
||||||
proxies: ' None '
|
|
||||||
LLM_MODEL: ' gpt-3.5-turbo '
|
|
||||||
AVAIL_LLM_MODELS: ' ["gpt-3.5-turbo", "gpt-4"] '
|
|
||||||
ENABLE_AUDIO: ' True '
|
|
||||||
LOCAL_MODEL_DEVICE: ' cuda '
|
|
||||||
DEFAULT_WORKER_NUM: ' 20 '
|
|
||||||
WEB_PORT: ' 12343 '
|
|
||||||
ADD_WAIFU: ' True '
|
|
||||||
THEME: ' Chuanhu-Small-and-Beautiful '
|
|
||||||
ALIYUN_APPKEY: ' RoP1ZrM84DnAFkZK '
|
|
||||||
ALIYUN_TOKEN: ' f37f30e0f9934c34a992f6f64f7eba4f '
|
|
||||||
# (无需填写) ALIYUN_ACCESSKEY: ' LTAI5q6BrFUzoRXVGUWnekh1 '
|
|
||||||
# (无需填写) ALIYUN_SECRET: ' eHmI20AVWIaQZ0CiTD2bGQVsaP9i68 '
|
|
||||||
|
|
||||||
# 「WEB_PORT暴露方法1: 适用于Linux」与宿主的网络融合
|
|
||||||
network_mode: "host"
|
|
||||||
|
|
||||||
# 启动命令
|
|
||||||
command: >
|
|
||||||
bash -c "python3 -u main.py"
|
|
||||||
@@ -1 +0,0 @@
|
|||||||
# 此Dockerfile不再维护，请前往docs/GithubAction+ChatGLM+Moss
|
|
||||||
@@ -1 +0,0 @@
|
|||||||
# 此Dockerfile不再维护，请前往docs/GithubAction+NoLocal+Latex
|
|
||||||
@@ -1,39 +0,0 @@
|
|||||||
# docker build -t gpt-academic-all-capacity -f docs/GithubAction+AllCapacity --network=host --build-arg http_proxy=http://localhost:10881 --build-arg https_proxy=http://localhost:10881 .
|
|
||||||
|
|
||||||
# 从NVIDIA源，从而支持显卡（检查宿主的nvidia-smi中的cuda版本必须>=11.3）
|
|
||||||
FROM fuqingxu/11.3.1-runtime-ubuntu20.04-with-texlive:latest
|
|
||||||
|
|
||||||
# edge-tts需要的依赖，某些pip包所需的依赖
|
|
||||||
RUN apt update && apt install ffmpeg build-essential -y
|
|
||||||
|
|
||||||
# use python3 as the system default python
|
|
||||||
WORKDIR /gpt
|
|
||||||
RUN curl -sS https://bootstrap.pypa.io/get-pip.py | python3.8
|
|
||||||
# 下载pytorch
|
|
||||||
RUN python3 -m pip install torch --extra-index-url https://download.pytorch.org/whl/cu113
|
|
||||||
# 准备pip依赖
|
|
||||||
RUN python3 -m pip install openai numpy arxiv rich
|
|
||||||
RUN python3 -m pip install colorama Markdown pygments pymupdf
|
|
||||||
RUN python3 -m pip install python-docx moviepy pdfminer
|
|
||||||
RUN python3 -m pip install zh_langchain==0.2.1 pypinyin
|
|
||||||
RUN python3 -m pip install rarfile py7zr
|
|
||||||
RUN python3 -m pip install aliyun-python-sdk-core==2.13.3 pyOpenSSL webrtcvad scipy git+https://github.com/aliyun/alibabacloud-nls-python-sdk.git
|
|
||||||
# 下载分支
|
|
||||||
WORKDIR /gpt
|
|
||||||
RUN git clone --depth=1 https://github.com/binary-husky/gpt_academic.git
|
|
||||||
WORKDIR /gpt/gpt_academic
|
|
||||||
RUN git clone --depth=1 https://github.com/OpenLMLab/MOSS.git request_llms/moss
|
|
||||||
|
|
||||||
RUN python3 -m pip install -r requirements.txt
|
|
||||||
RUN python3 -m pip install -r request_llms/requirements_moss.txt
|
|
||||||
RUN python3 -m pip install -r request_llms/requirements_qwen.txt
|
|
||||||
RUN python3 -m pip install -r request_llms/requirements_chatglm.txt
|
|
||||||
RUN python3 -m pip install -r request_llms/requirements_newbing.txt
|
|
||||||
RUN python3 -m pip install nougat-ocr
|
|
||||||
|
|
||||||
|
|
||||||
# 预热Tiktoken模块
|
|
||||||
RUN python3 -c 'from check_proxy import warm_up_modules; warm_up_modules()'
|
|
||||||
|
|
||||||
# 启动
|
|
||||||
CMD ["python3", "-u", "main.py"]
|
|
||||||
@@ -1,31 +0,0 @@
|
|||||||
|
|
||||||
# 从NVIDIA源，从而支持显卡运损（检查宿主的nvidia-smi中的cuda版本必须>=11.3）
|
|
||||||
FROM nvidia/cuda:11.3.1-runtime-ubuntu20.04
|
|
||||||
RUN apt-get update
|
|
||||||
RUN apt-get install -y curl proxychains curl gcc
|
|
||||||
RUN apt-get install -y git python python3 python-dev python3-dev --fix-missing
|
|
||||||
|
|
||||||
# edge-tts需要的依赖，某些pip包所需的依赖
|
|
||||||
RUN apt update && apt install ffmpeg build-essential -y
|
|
||||||
|
|
||||||
# use python3 as the system default python
|
|
||||||
RUN curl -sS https://bootstrap.pypa.io/get-pip.py | python3.8
|
|
||||||
# 下载pytorch
|
|
||||||
RUN python3 -m pip install torch --extra-index-url https://download.pytorch.org/whl/cu113
|
|
||||||
# 下载分支
|
|
||||||
WORKDIR /gpt
|
|
||||||
RUN git clone --depth=1 https://github.com/binary-husky/gpt_academic.git
|
|
||||||
WORKDIR /gpt/gpt_academic
|
|
||||||
RUN git clone https://github.com/OpenLMLab/MOSS.git request_llms/moss
|
|
||||||
RUN python3 -m pip install -r requirements.txt
|
|
||||||
RUN python3 -m pip install -r request_llms/requirements_moss.txt
|
|
||||||
RUN python3 -m pip install -r request_llms/requirements_qwen.txt
|
|
||||||
RUN python3 -m pip install -r request_llms/requirements_chatglm.txt
|
|
||||||
RUN python3 -m pip install -r request_llms/requirements_newbing.txt
|
|
||||||
|
|
||||||
|
|
||||||
# 预热Tiktoken模块
|
|
||||||
RUN python3 -c 'from check_proxy import warm_up_modules; warm_up_modules()'
|
|
||||||
|
|
||||||
# 启动
|
|
||||||
CMD ["python3", "-u", "main.py"]
|
|
||||||
@@ -1,37 +0,0 @@
|
|||||||
# 从NVIDIA源，从而支持显卡运损（检查宿主的nvidia-smi中的cuda版本必须>=11.3）
|
|
||||||
FROM nvidia/cuda:11.3.1-runtime-ubuntu20.04
|
|
||||||
ARG useProxyNetwork=''
|
|
||||||
RUN apt-get update
|
|
||||||
RUN apt-get install -y curl proxychains curl g++
|
|
||||||
RUN apt-get install -y git python python3 python-dev python3-dev --fix-missing
|
|
||||||
|
|
||||||
# use python3 as the system default python
|
|
||||||
RUN curl -sS https://bootstrap.pypa.io/get-pip.py | python3.8
|
|
||||||
|
|
||||||
# 下载pytorch
|
|
||||||
RUN python3 -m pip install torch --extra-index-url https://download.pytorch.org/whl/cu113
|
|
||||||
|
|
||||||
# 下载分支
|
|
||||||
WORKDIR /gpt
|
|
||||||
RUN git clone --depth=1 https://github.com/binary-husky/gpt_academic.git
|
|
||||||
WORKDIR /gpt/gpt_academic
|
|
||||||
RUN python3 -m pip install -r requirements.txt
|
|
||||||
RUN python3 -m pip install -r request_llms/requirements_chatglm.txt
|
|
||||||
RUN python3 -m pip install -r request_llms/requirements_newbing.txt
|
|
||||||
RUN python3 -m pip install -r request_llms/requirements_jittorllms.txt -i https://pypi.jittor.org/simple -I
|
|
||||||
|
|
||||||
# 下载JittorLLMs
|
|
||||||
RUN git clone https://github.com/binary-husky/JittorLLMs.git --depth 1 request_llms/jittorllms
|
|
||||||
|
|
||||||
# edge-tts需要的依赖
|
|
||||||
RUN apt update && apt install ffmpeg -y
|
|
||||||
|
|
||||||
# 禁用缓存，确保更新代码
|
|
||||||
ADD "https://www.random.org/cgi-bin/randbyte?nbytes=10&format=h" skipcache
|
|
||||||
RUN git pull
|
|
||||||
|
|
||||||
# 预热Tiktoken模块
|
|
||||||
RUN python3 -c 'from check_proxy import warm_up_modules; warm_up_modules()'
|
|
||||||
|
|
||||||
# 启动
|
|
||||||
CMD ["python3", "-u", "main.py"]
|
|
||||||
@@ -1,22 +0,0 @@
|
|||||||
# 此Dockerfile适用于“无本地模型”的环境构建，如果需要使用chatglm等本地模型，请参考 docs/Dockerfile+ChatGLM
|
|
||||||
# 如何构建: 先修改 `config.py`， 然后 docker build -t gpt-academic-nolocal -f docs/Dockerfile+NoLocal .
|
|
||||||
# 如何运行: docker run --rm -it --net=host gpt-academic-nolocal
|
|
||||||
FROM python:3.11
|
|
||||||
|
|
||||||
# 指定路径
|
|
||||||
WORKDIR /gpt
|
|
||||||
|
|
||||||
# 装载项目文件
|
|
||||||
COPY . .
|
|
||||||
|
|
||||||
# 安装依赖
|
|
||||||
RUN pip3 install -r requirements.txt
|
|
||||||
|
|
||||||
# edge-tts需要的依赖
|
|
||||||
RUN apt update && apt install ffmpeg -y
|
|
||||||
|
|
||||||
# 可选步骤，用于预热模块
|
|
||||||
RUN python3 -c 'from check_proxy import warm_up_modules; warm_up_modules()'
|
|
||||||
|
|
||||||
# 启动
|
|
||||||
CMD ["python3", "-u", "main.py"]
|
|
||||||
@@ -1,25 +0,0 @@
|
|||||||
# 此Dockerfile适用于“无本地模型”的环境构建，如果需要使用chatglm等本地模型，请参考 docs/Dockerfile+ChatGLM
|
|
||||||
# 如何构建: 先修改 `config.py`， 然后 docker build -t gpt-academic-nolocal -f docs/Dockerfile+NoLocal .
|
|
||||||
# 如何运行: docker run --rm -it --net=host gpt-academic-nolocal
|
|
||||||
FROM python:3.11
|
|
||||||
|
|
||||||
# 指定路径
|
|
||||||
WORKDIR /gpt
|
|
||||||
|
|
||||||
# 装载项目文件
|
|
||||||
COPY . .
|
|
||||||
|
|
||||||
# 安装依赖
|
|
||||||
RUN pip3 install -r requirements.txt
|
|
||||||
|
|
||||||
# 安装语音插件的额外依赖
|
|
||||||
RUN pip3 install aliyun-python-sdk-core==2.13.3 pyOpenSSL webrtcvad scipy git+https://github.com/aliyun/alibabacloud-nls-python-sdk.git
|
|
||||||
|
|
||||||
# edge-tts需要的依赖
|
|
||||||
RUN apt update && apt install ffmpeg -y
|
|
||||||
|
|
||||||
# 可选步骤，用于预热模块
|
|
||||||
RUN python3 -c 'from check_proxy import warm_up_modules; warm_up_modules()'
|
|
||||||
|
|
||||||
# 启动
|
|
||||||
CMD ["python3", "-u", "main.py"]
|
|
||||||
@@ -1,34 +0,0 @@
|
|||||||
# 此Dockerfile适用于"无本地模型"的环境构建，如果需要使用chatglm等本地模型，请参考 docs/Dockerfile+ChatGLM
|
|
||||||
# - 1 修改 `config.py`
|
|
||||||
# - 2 构建 docker build -t gpt-academic-nolocal-latex -f docs/GithubAction+NoLocal+Latex .
|
|
||||||
# - 3 运行 docker run -v /home/fuqingxu/arxiv_cache:/root/arxiv_cache --rm -it --net=host gpt-academic-nolocal-latex
|
|
||||||
|
|
||||||
FROM menghuan1918/ubuntu_uv_ctex:latest
|
|
||||||
ENV DEBIAN_FRONTEND=noninteractive
|
|
||||||
SHELL ["/bin/bash", "-c"]
|
|
||||||
WORKDIR /gpt
|
|
||||||
|
|
||||||
# 先复制依赖文件
|
|
||||||
COPY requirements.txt .
|
|
||||||
|
|
||||||
# 安装依赖
|
|
||||||
RUN pip install --break-system-packages openai numpy arxiv rich colorama Markdown pygments pymupdf python-docx pdfminer \
|
|
||||||
&& pip install --break-system-packages -r requirements.txt \
|
|
||||||
&& if [ "$(uname -m)" = "x86_64" ]; then \
|
|
||||||
pip install --break-system-packages nougat-ocr; \
|
|
||||||
fi \
|
|
||||||
&& pip cache purge \
|
|
||||||
&& rm -rf /root/.cache/pip/*
|
|
||||||
|
|
||||||
# 创建非root用户
|
|
||||||
RUN useradd -m gptuser && chown -R gptuser /gpt
|
|
||||||
USER gptuser
|
|
||||||
|
|
||||||
# 最后才复制代码文件,这样代码更新时只需重建最后几层，可以大幅减少docker pull所需的大小
|
|
||||||
COPY --chown=gptuser:gptuser . .
|
|
||||||
|
|
||||||
# 可选步骤，用于预热模块
|
|
||||||
RUN python3 -c 'from check_proxy import warm_up_modules; warm_up_modules()'
|
|
||||||
|
|
||||||
# 启动
|
|
||||||
CMD ["python3", "-u", "main.py"]
|
|
||||||
@@ -1,29 +0,0 @@
|
|||||||
# 此Dockerfile适用于“无本地模型”的环境构建，如果需要使用chatglm等本地模型，请参考 docs/Dockerfile+ChatGLM
|
|
||||||
# 如何构建: 先修改 `config.py`， 然后 docker build -t gpt-academic-nolocal-vs -f docs/GithubAction+NoLocal+Vectordb .
|
|
||||||
# 如何运行: docker run --rm -it --net=host gpt-academic-nolocal-vs
|
|
||||||
FROM python:3.11
|
|
||||||
|
|
||||||
# 指定路径
|
|
||||||
WORKDIR /gpt
|
|
||||||
|
|
||||||
# 装载项目文件
|
|
||||||
COPY . .
|
|
||||||
|
|
||||||
# 安装依赖
|
|
||||||
RUN pip3 install -r requirements.txt
|
|
||||||
|
|
||||||
# 安装知识库插件的额外依赖
|
|
||||||
RUN apt-get update && apt-get install libgl1 -y
|
|
||||||
RUN pip3 install torch torchvision --index-url https://download.pytorch.org/whl/cpu
|
|
||||||
RUN pip3 install transformers protobuf langchain sentence-transformers faiss-cpu nltk beautifulsoup4 bitsandbytes tabulate icetk --upgrade
|
|
||||||
RUN pip3 install unstructured[all-docs] --upgrade
|
|
||||||
RUN python3 -c 'from check_proxy import warm_up_vectordb; warm_up_vectordb()'
|
|
||||||
|
|
||||||
# edge-tts需要的依赖
|
|
||||||
RUN apt update && apt install ffmpeg -y
|
|
||||||
|
|
||||||
# 可选步骤，用于预热模块
|
|
||||||
RUN python3 -c 'from check_proxy import warm_up_modules; warm_up_modules()'
|
|
||||||
|
|
||||||
# 启动
|
|
||||||
CMD ["python3", "-u", "main.py"]
|
|
||||||
@@ -1,343 +0,0 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
> **ملحوظة**
|
|
||||||
>
|
|
||||||
> تمت ترجمة هذا الملف README باستخدام GPT (بواسطة المكون الإضافي لهذا المشروع) وقد لا تكون الترجمة 100٪ موثوقة، يُرجى التمييز بعناية بنتائج الترجمة.
|
|
||||||
>
|
|
||||||
> 2023.11.7: عند تثبيت التبعيات، يُرجى اختيار الإصدار المُحدد في `requirements.txt`. الأمر للتثبيت: `pip install -r requirements.txt`.
|
|
||||||
|
|
||||||
# <div align=center><img src="logo.png" width="40"> GPT الأكاديمي</div>
|
|
||||||
|
|
||||||
**إذا كنت تحب هذا المشروع، فيُرجى إعطاؤه Star. لترجمة هذا المشروع إلى لغة عشوائية باستخدام GPT، قم بقراءة وتشغيل [`multi_language.py`](multi_language.py) (تجريبي).
|
|
||||||
|
|
||||||
> **ملحوظة**
|
|
||||||
>
|
|
||||||
> 1. يُرجى ملاحظة أنها الإضافات (الأزرار) المميزة فقط التي تدعم قراءة الملفات، وبعض الإضافات توجد في قائمة منسدلة في منطقة الإضافات. بالإضافة إلى ذلك، نرحب بأي Pull Request جديد بأعلى أولوية لأي إضافة جديدة.
|
|
||||||
>
|
|
||||||
> 2. تُوضّح كل من الملفات في هذا المشروع وظيفتها بالتفصيل في [تقرير الفهم الذاتي `self_analysis.md`](https://github.com/binary-husky/gpt_academic/wiki/GPT‐Academic项目自译解报告). يمكنك في أي وقت أن تنقر على إضافة وظيفة ذات صلة لاستدعاء GPT وإعادة إنشاء تقرير الفهم الذاتي للمشروع. للأسئلة الشائعة [`الويكي`](https://github.com/binary-husky/gpt_academic/wiki). [طرق التثبيت العادية](#installation) | [نصب بنقرة واحدة](https://github.com/binary-husky/gpt_academic/releases) | [تعليمات التكوين](https://github.com/binary-husky/gpt_academic/wiki/项目配置说明).
|
|
||||||
>
|
|
||||||
> 3. يتم توافق هذا المشروع مع ودعم توصيات اللغة البيجائية الأكبر شمولًا وشجاعة لمثل ChatGLM. يمكنك توفير العديد من مفاتيح Api المشتركة في تكوين الملف، مثل `API_KEY="openai-key1,openai-key2,azure-key3,api2d-key4"`. عند تبديل مؤقت لـ `API_KEY`، قم بإدخال `API_KEY` المؤقت في منطقة الإدخال ثم اضغط على زر "إدخال" لجعله ساري المفعول.
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
<div align="center">
|
|
||||||
|
|
||||||
الوظائف (⭐= وظائف مُضافة حديثًا) | الوصف
|
|
||||||
--- | ---
|
|
||||||
⭐[التوصل لنموذج جديد](https://github.com/binary-husky/gpt_academic/wiki/%E5%A6%82%E4%BD%95%E5%88%87%E6%8D%A2%E6%A8%A1%E5%9E%8B)! | بحث بيدو[تشيان فان](https://cloud.baidu.com/doc/WENXINWORKSHOP/s/Nlks5zkzu) ووينسين[جينرال](https://modelscope.cn/models/qwen/Qwen-7B-Chat/summary)، مختبرات شنغهاي للذكاء الصناعي[شو شينغ](https://github.com/InternLM/InternLM)، إكسنفلام[زينغهو]https://xinghuo.xfyun.cn/)، [LLaMa2](https://huggingface.co/meta-llama/Llama-2-7b-chat-hf)، واجهة بيانية ذكية و3 خدمات إضافية [DALLE3]
|
|
||||||
الجودة الفائقة، الترجمة، شرح الكود | الإصلاح الفوري للاخطاء النحوية في الأبحاث وترجمة وتحسين التصريف اللغوي للأكواد
|
|
||||||
[اختصارات مخصصة](https://www.bilibili.com/video/BV14s4y1E7jN) | دعم الاختصارات المخصصة
|
|
||||||
تصميم قابل للتوسيع | دعم الإضافات القوية المخصصة (الوظائف)، الإضافات قابلة للتحديث بشكل فوري
|
|
||||||
[تحليل البرنامج](https://www.bilibili.com/video/BV1cj411A7VW) | [وظائف] التحليل الشجري بناءً على البرنامج من Python/C/C++/Java/Lua/..., أو [التحليل الذاتي](https://www.bilibili.com/video/BV1cj411A7VW)
|
|
||||||
قراءة وترجمة الأبحاث | [وظائف] فك تشفير كامل لأوراق البحث بتنسيق LaTeX/PDF وإنشاء مستخلص
|
|
||||||
ترجمة وتحسين أوراق اللاتكس | [وظائف] ترجمة أو تحسين الأوراق المكتوبة بلاتكس
|
|
||||||
إنشاء تعليقات الدوال دفعة واحدة | [وظائف] إنشاء تعليقات الدوال بدفعة واحدة
|
|
||||||
ترجمة Markdown بين اللغتين العربية والإنجليزية | [وظائف] هل رأيت الـ 5 لغات المستخدمة في منشور [README](https://github.com/binary-husky/gpt_academic/blob/master/docs/README_EN.md) ؟
|
|
||||||
إنشاء تقرير تحليل الدردشة | [وظائف] إنشاء تقرير ملخص بعد تشغيله
|
|
||||||
ترجمة كاملة لأوراق PDF | [وظائف] تحليل الأوراق بتنسيق PDF لتحديد العنوان وملخصها وترجمتها (متعدد الخيوط)
|
|
||||||
مساعدة Arxiv | [وظائف] قم بإدخال رابط مقال Arxiv لترجمة الملخص وتحميل ملف PDF
|
|
||||||
تصحيح لاتكس بضغطة زر واحدة | [وظائف] إكمال تصحيح لاتكس بناءً على التركيبة النحوية، إخراج همز المقابل للمقارنة PDF
|
|
||||||
مساعد بحث Google بنسخة محلية | [وظائف] قم بتقديم رابط لصفحة بحث Google Scholar العشوائي حتى يساعدك GPT في كتابة [الأبحاث المتعلقة](https://www.bilibili.com/video/BV1GP411U7Az/)
|
|
||||||
تجميع معلومات الويب + GPT | [وظائف] جمع المعلومات من الويب بشكل سهل للرد على الأسئلة لجعل المعلومات محدثة باستمرار
|
|
||||||
⭐ترجمة دقيقة لأوراق Arxiv ([Docker](https://github.com/binary-husky/gpt_academic/pkgs/container/gpt_academic_with_latex)) | [وظائف] ترجمة مقالات Arxiv عالية الجودة بنقرة واحدة، أفضل أداة حاليا للترجمة
|
|
||||||
⭐[إدخال الصوت الفوري](https://github.com/binary-husky/gpt_academic/blob/master/docs/use_audio.md) | [وظائف] (غير متزامن) استماع الصوت وقطعه تلقائيًا وتحديد وقت الإجابة تلقائيًا
|
|
||||||
عرض الصيغ/الصور/الجداول | يمكن عرض الصيغ بشكل [TEX](https://user-images.githubusercontent.com/96192199/230598842-1d7fcddd-815d-40ee-af60-baf488a199df.png) وأيضًا بتنسيق رسومي، يدعم عرض الصيغ وإبراز الكود
|
|
||||||
⭐إضغط على وكيل "شارلوت الذكي" | [وظائف] استكمال الذكاء للكأس الأول للذكاء المكتسب من مايكروسوفت، اكتشاف وتطوير عالمي العميل
|
|
||||||
تبديل الواجهة المُظلمة | يمكنك التبديل إلى الواجهة المظلمة بإضافة ```/?__theme=dark``` إلى نهاية عنوان URL في المتصفح
|
|
||||||
دعم المزيد من نماذج LLM | دعم لجميع GPT3.5 وGPT4 و[ChatGLM2 في جامعة ثوه في لين](https://github.com/THUDM/ChatGLM2-6B) و[MOSS في جامعة فودان](https://github.com/OpenLMLab/MOSS)
|
|
||||||
⭐تحوي انطباعة "ChatGLM2" | يدعم استيراد "ChatGLM2" ويوفر إضافة المساعدة في تعديله
|
|
||||||
دعم المزيد من نماذج "LLM"، دعم [نشر الحديس](https://huggingface.co/spaces/qingxu98/gpt-academic) | انضم إلى واجهة "Newbing" (Bing الجديدة)،نقدم نماذج Jittorllms الجديدة تؤيدهم [LLaMA](https://github.com/facebookresearch/llama) و [盘古α](https://openi.org.cn/pangu/)
|
|
||||||
⭐حزمة "void-terminal" للشبكة (pip) | قم بطلب كافة وظائف إضافة هذا المشروع في python بدون واجهة رسومية (قيد التطوير)
|
|
||||||
⭐PCI-Express لإعلام (PCI) | [وظائف] باللغة الطبيعية، قم بتنفيذ المِهام الأخرى في المشروع
|
|
||||||
المزيد من العروض (إنشاء الصور وغيرها)……| شاهد أكثر في نهاية هذا المستند ...
|
|
||||||
</div>
|
|
||||||
|
|
||||||
|
|
||||||
- شكل جديد (عن طريق تعديل الخيار LAYOUT في `config.py` لقانون التوزيع "اليمين أو اليسار" أو "الأعلى أو الأسفل")
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://github.com/binary-husky/gpt_academic/assets/96192199/d81137c3-affd-4cd1-bb5e-b15610389762" width="700" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
|
|
||||||
- جميع الأزرار يتم إنشاؤها ديناميكيًا من خلال قراءة functional.py ويمكن إضافة وظائف مخصصة بحرية وتحرير الحافظة
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/231975334-b4788e91-4887-412f-8b43-2b9c5f41d248.gif" width="700" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
- التجميل / التحوير
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/231980294-f374bdcb-3309-4560-b424-38ef39f04ebd.gif" width="700" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
- إذا تضمّن الإخراج معادلات، فسيتم عرضها بشكلٍ يمكّن من النسخ والقراءة على النحوين: TEX ورسومية.
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/230598842-1d7fcddd-815d-40ee-af60-baf488a199df.png" width="700" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
- هل تشعر بالكسل من قراءة كود المشروع؟ قم بمدها مباشرةً إلى ChatGPT
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/226935232-6b6a73ce-8900-4aee-93f9-733c7e6fef53.png" width="700" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
- دمج نماذج اللغات الكبيرة المختلفة (ChatGLM + OpenAI-GPT3.5 + [API2D](https://api2d.com/)-GPT4)
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/232537274-deca0563-7aa6-4b5d-94a2-b7c453c47794.png" width="700" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
# Installation
|
|
||||||
### طريقة التثبيت الأولى: التشغيل المباشر (Windows، Linux أو MacOS)
|
|
||||||
|
|
||||||
1. قم بتنزيل المشروع
|
|
||||||
```sh
|
|
||||||
git clone --depth=1 https://github.com/binary-husky/gpt_academic.git
|
|
||||||
cd gpt_academic
|
|
||||||
```
|
|
||||||
|
|
||||||
2. قم بتكوين لغة البرمجة Python
|
|
||||||
|
|
||||||
في ملف `config.py`، قم بتكوين مفتاح الواجهة API والإعدادات الأخرى، [انقر هنا للاطلاع على طريقة تكوين الإعدادات في بيئة شبكة خاصة](https://github.com/binary-husky/gpt_academic/issues/1). [انقر هنا لزيارة صفحة الويكي](https://github.com/binary-husky/gpt_academic/wiki/توضيحات-تكوين-المشروع).
|
|
||||||
|
|
||||||
" ستقوم البرنامج بفحص وجود ملف تكوين خاص يسمى `config_private.py` بأولوية، وسيستخدم التكوينات الموجودة فيه لتجاوز التكوينات ذات الأسماء المطابقة في `config.py`. إذا كنت تفهم هذه الطريقة ونظام القراءة، فإننا نوصي بشدة بإنشاء ملف تكوين جديد يسمى `config_private.py` بجوار `config.py` ونقل (نسخ) التكوينات الموجودة في `config.py` إلى `config_private.py` (يجب نسخ العناصر التي قمت بتعديلها فقط). "
|
|
||||||
|
|
||||||
" يدعم المشروع التكوين من خلال `المتغيرات المحيطية`، ويمكن تحديد تنسيق كتابة المتغيرات المحيطية من خلال ملف `docker-compose.yml` أو صفحة الويكي الخاصة بنا. تعتمد أولوية القراءة على التكوينات على التالي: `المتغيرات المحيطية` > `config_private.py` > `config.py`. "
|
|
||||||
|
|
||||||
3. قم بتثبيت التبعيات
|
|
||||||
```sh
|
|
||||||
# (الخيار الأول: إذا كنت تعرف Python، python>=3.9) الملحوظة: استخدم مستودع pip الرسمي أو مستودع pip آلي بباي، يمكن تغيير المستودع المؤقت بواسطة الأمر: python -m pip install -r requirements.txt -i https://mirrors.aliyun.com/pypi/simple/
|
|
||||||
python -m pip install -r requirements.txt
|
|
||||||
|
|
||||||
# (الخيار الثاني: باستخدام Anaconda) الخطوات مشابهة (https://www.bilibili.com/video/BV1rc411W7Dr):
|
|
||||||
conda create -n gptac_venv python=3.11 # إنشاء بيئة Anaconda
|
|
||||||
conda activate gptac_venv # تنشيط بيئة Anaconda
|
|
||||||
python -m pip install -r requirements.txt # هذه الخطوة مطابقة لخطوة تثبيت pip
|
|
||||||
```
|
|
||||||
|
|
||||||
|
|
||||||
<details><summary>إذا كنت بحاجة إلى دعم ChatGLM2 من الجامعة الصينية للاقتصاد وإدارة الأعمال وموس من جامعة فودان كخادم وجودة عالية لطرح الأسئلة، انقر هنا للعرض</summary>
|
|
||||||
<p>
|
|
||||||
|
|
||||||
【خطوات اختيارية】إذا كنت بحاجة إلى دعم جودة عالية لتشات جامعة تسينهوا (ChatGLM2) الصينية وجامعة فودان (MOSS)، يتعين عليك تثبيت تبعيات إضافية (شرط مسبق: التعامل مع Python واستخدام Pytorch وتوفر الحاسوب الشخصي بمواصفات قوية):
|
|
||||||
```sh
|
|
||||||
# 【خطوات اختيارية 1】دعم جودة عالية لتشات جامعة تسينهوا (ChatGLM2)
|
|
||||||
python -m pip install -r request_llms/requirements_chatglm.txt
|
|
||||||
|
|
||||||
# 【خطوات اختيارية 2】دعم جودة عالية لتشات جامعة فودان (MOSS)
|
|
||||||
python -m pip install -r request_llms/requirements_moss.txt
|
|
||||||
git clone --depth=1 https://github.com/OpenLMLab/MOSS.git request_llms/moss # عند تنفيذ هذا الأمر، يجب أن تكون في مسار المشروع الرئيسي
|
|
||||||
|
|
||||||
# 【خطوات اختيارية 3】دعم RWKV Runner
|
|
||||||
راجع الويكي: https://github.com/binary-husky/gpt_academic/wiki/دليل-تكوين-RWKV
|
|
||||||
|
|
||||||
# 【خطوات اختيارية 4】تأكد من أن ملف التكوين config.py يحتوي على النماذج المرجوة، وهناك النماذج المدعومة حاليًا التالية (توجد خطط لتشغيل "jittorllms" في docker فقط):
|
|
||||||
AVAIL_LLM_MODELS = ["gpt-3.5-turbo", "api2d-gpt-3.5-turbo", "gpt-4", "api2d-gpt-4", "chatglm", "moss"] # + ["jittorllms_rwkv", "jittorllms_pangualpha", "jittorllms_llama"]
|
|
||||||
```
|
|
||||||
|
|
||||||
</p>
|
|
||||||
</details>
|
|
||||||
|
|
||||||
|
|
||||||
4. تشغيل البرنامج
|
|
||||||
```sh
|
|
||||||
python main.py
|
|
||||||
```
|
|
||||||
|
|
||||||
### طريقة التثبيت الثانية: استخدام Docker
|
|
||||||
|
|
||||||
0. نصب القدرات الكاملة للمشروع (هذا هو الصورة الكبيرة التي تحتوي على CUDA و LaTeX. ولكن إذا كانت سرعة الإنترنت بطيئة أو القرص الصلب صغير، فإننا لا نوصي باستخدام هذا الخيار)
|
|
||||||
[](https://github.com/binary-husky/gpt_academic/actions/workflows/build-with-all-capacity.yml)
|
|
||||||
|
|
||||||
``` sh
|
|
||||||
# قم بتعديل ملف docker-compose.yml للحفاظ على الخطة رقم 0 وحذف الخطط الأخرى. ثم أشغل:
|
|
||||||
docker-compose up
|
|
||||||
```
|
|
||||||
|
|
||||||
1. تشغيل نموذج ChatGPT فقط + 文心一言 (Wenxin YIYan) + Spark عبر الإنترنت (يُوصى بهذا الخيار للمعظم)
|
|
||||||
|
|
||||||
[](https://github.com/binary-husky/gpt_academic/actions/workflows/build-without-local-llms.yml)
|
|
||||||
[](https://github.com/binary-husky/gpt_academic/actions/workflows/build-with-latex.yml)
|
|
||||||
[](https://github.com/binary-husky/gpt_academic/actions/workflows/build-with-audio-assistant.yml)
|
|
||||||
|
|
||||||
``` sh
|
|
||||||
# قم بتعديل ملف docker-compose.yml للحفاظ على الخطة رقم 1 وحذف الخطط الأخرى. ثم أشغل:
|
|
||||||
docker-compose up
|
|
||||||
```
|
|
||||||
|
|
||||||
P.S. للاستفادة من إمكانية اللافتكس الإضافية، يرجى الرجوع إلى الويكي. بالإضافة إلى ذلك، يمكنك استخدام الخطة 4 أو الخطة 0 مباشرة للحصول على إمكانية اللافتكس.
|
|
||||||
|
|
||||||
2. تشغيل نموذج ChatGPT + نموذج ChatGLM2 + نموذج MOSS + نموذج LLAMA2 + تون يي تشين ون (QiChaYiWen) (يتطلب معرفة بتشغيل نيفيديا دوكر (Nvidia Docker))
|
|
||||||
|
|
||||||
[](https://github.com/binary-husky/gpt_academic/actions/workflows/build-with-chatglm.yml)
|
|
||||||
|
|
||||||
``` sh
|
|
||||||
# قم بتعديل ملف docker-compose.yml للحفاظ على الخطة رقم 2 وحذف الخطط الأخرى. ثم أشغل:
|
|
||||||
docker-compose up
|
|
||||||
```
|
|
||||||
|
|
||||||
### طريقة التثبيت الثالثة: طرائق نشر أخرى
|
|
||||||
1. **نصوص بنقرة واحدة لأنظمة Windows**.
|
|
||||||
يمكن لمستخدمي Windows الذين لا يعرفون بيئة Python تنزيل سكربت التشغيل بنقرة واحدة من [الإصدارات](https://github.com/binary-husky/gpt_academic/releases) المنشورة لتثبيت الإصدار الذي لا يحتوي على نماذج محلية.
|
|
||||||
المساهمة في السكربت تعود لـ[oobabooga](https://github.com/oobabooga/one-click-installers).
|
|
||||||
|
|
||||||
2. استخدام واجهة برمجة تطبيقات (API) مطراف ثالثة، Microsoft Azure، ونشوة النص، وغيرها، يرجى الرجوع إلى [صفحة الويكي](https://github.com/binary-husky/gpt_academic/wiki/إعدادات-التكوين-للمشروع) الخاصة بنا
|
|
||||||
|
|
||||||
3. دليل تجنب المشاكل عند نشر المشروع في خوادم السحابة.
|
|
||||||
يرجى زيارة صفحة [دليل نشر خوادم السحابة في المحيط](https://github.com/binary-husky/gpt_academic/wiki/دليل-نشر-خوادم-السحابة)
|
|
||||||
|
|
||||||
4. طرائق نشر المشروع بأحدث الأساليب
|
|
||||||
- استخدام Sealos للنشر السريع [بنقرة واحدة](https://github.com/binary-husky/gpt_academic/issues/993).
|
|
||||||
- استخدم WSL2 (Windows Subsystem for Linux). يُرجى زيارة صفحة الويكي [لدليل التثبيت-2](https://github.com/binary-husky/gpt_academic/wiki/دليل-تشغيل-WSL2-(Windows-Subsystem-for-Linux)
|
|
||||||
- كيفية تشغيل البرنامج تحت عنوان فرعي (على سبيل المثال: `http://localhost/subpath`). يُرجى زيارة [إرشادات FastAPI](docs/WithFastapi.md)
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
# الاستخدام المتقدم
|
|
||||||
### I: إنشاء أزرار مخصصة (اختصارات أكاديمية)
|
|
||||||
افتح أي محرر نصوص وافتح `core_functional.py` وأضف الإدخالات التالية ثم أعد تشغيل البرنامج. (إذا كانت الأزرار موجودة بالفعل، بإمكانك تعديل البادئة واللاحقة حراريًا دون الحاجة لإعادة تشغيل البرنامج)
|
|
||||||
على سبيل المثال:
|
|
||||||
```
|
|
||||||
"ترجمة سوبر الإنجليزية إلى العربية": {
|
|
||||||
# البادئة، ستتم إضافتها قبل إدخالاتك. مثلاً، لوصف ما تريده مثل ترجمة أو شرح كود أو تلوين وهلم جرا
|
|
||||||
"بادئة": "يرجى ترجمة النص التالي إلى العربية ثم استخدم جدول Markdown لشرح المصطلحات المختصة المذكورة في النص:\n\n",
|
|
||||||
|
|
||||||
# اللاحقة، سيتم إضافتها بعد إدخالاتك. يمكن استخدامها لوضع علامات اقتباس حول إدخالك.
|
|
||||||
"لاحقة": "",
|
|
||||||
},
|
|
||||||
```
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/226899272-477c2134-ed71-4326-810c-29891fe4a508.png" width="500" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
### II: إنشاء مكونات وظيفية مخصصة
|
|
||||||
قم بكتابة مكونات وظيفية قوية لتنفيذ أي مهمة ترغب في الحصول عليها وحتى تلك التي لم تخطر لك على بال.
|
|
||||||
إن إنشاء وتصحيح المكونات في هذا المشروع سهل للغاية، فما عليك سوى أن تمتلك بعض المعرفة الأساسية في لغة البرمجة بايثون وتستند على القالب الذي نقدمه.
|
|
||||||
للمزيد من التفاصيل، يُرجى الاطلاع على [دليل المكونات الوظيفية](https://github.com/binary-husky/gpt_academic/wiki/%E5%87%BD%E6%95%B0%E6%8F%92%E4%BB%B6%E6%8C%87%E5%8D%97).
|
|
||||||
|
|
||||||
|
|
||||||
# التحديثات
|
|
||||||
### I: تحديثات
|
|
||||||
|
|
||||||
1. ميزة حفظ الدردشة: يمكن حفظ الدردشة الحالية كملف HTML قابل للقراءة والاسترداد ببساطة عند استدعاء الوظيفة في منطقة المكونات `حفظ الدردشة الحالية` ، ويمكن استرجاع المحادثة السابقة ببساطة عند استدعاء الوظيفة في منطقة المكونات (القائمة المنسدلة) `تحميل سجل الدردشة` .
|
|
||||||
نصيحة: يمكنك النقر المباشر على `تحميل سجل الدردشة` بدون تحديد ملف لعرض ذاكرة التخزين المؤقت لسجلات HTML.
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/235222390-24a9acc0-680f-49f5-bc81-2f3161f1e049.png" width="500" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
2. ميزة ترجمة المقالات العلمية بواسطة Latex/Arxiv
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://github.com/binary-husky/gpt_academic/assets/96192199/002a1a75-ace0-4e6a-94e2-ec1406a746f1" height="250" > ===>
|
|
||||||
<img src="https://github.com/binary-husky/gpt_academic/assets/96192199/9fdcc391-f823-464f-9322-f8719677043b" height="250" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
3. محطة فراغ (فهم نغمة المستخدم من داخل اللغة الطبيعية واستدعاء وظائف أخرى تلقائيًا)
|
|
||||||
|
|
||||||
- الخطوة 1: اكتب "بالرجاء استدعاء وظيفة ترجمة المقالة الأكاديمية من PDF وعنوان المقال هو https://openreview.net/pdf?id=rJl0r3R9KX".
|
|
||||||
- الخطوة 2: انقر فوق "محطة الفراغ".
|
|
||||||
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://github.com/binary-husky/gpt_academic/assets/96192199/66f1b044-e9ff-4eed-9126-5d4f3668f1ed" width="500" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
4. تصميم الوظائف المتعددة القادرة على توفير وظائف قوية بواجهات بسيطة
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/229288270-093643c1-0018-487a-81e6-1d7809b6e90f.png" height="400" >
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/227504931-19955f78-45cd-4d1c-adac-e71e50957915.png" height="400" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
5. ترجمة وإلغاء ترجمة المشاريع الأخرى مفتوحة المصدر
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/226935232-6b6a73ce-8900-4aee-93f9-733c7e6fef53.png" height="250" >
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/226969067-968a27c1-1b9c-486b-8b81-ab2de8d3f88a.png" height="250" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
6. ميزة تزيين [live2d](https://github.com/fghrsh/live2d_demo) (مغلقة بشكل افتراضي، يتطلب تعديل `config.py`)
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/236432361-67739153-73e8-43fe-8111-b61296edabd9.png" width="500" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
7. إنتاج الصور من OpenAI
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://github.com/binary-husky/gpt_academic/assets/96192199/bc7ab234-ad90-48a0-8d62-f703d9e74665" width="500" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
8. تحليل وإجماع الصوت من OpenAI
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://github.com/binary-husky/gpt_academic/assets/96192199/709ccf95-3aee-498a-934a-e1c22d3d5d5b" width="500" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
9. إصلاح أخطاء اللغة الطبيعة في Latex
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://github.com/binary-husky/gpt_academic/assets/96192199/651ccd98-02c9-4464-91e1-77a6b7d1b033" height="200" > ===>
|
|
||||||
<img src="https://github.com/binary-husky/gpt_academic/assets/96192199/476f66d9-7716-4537-b5c1-735372c25adb" height="200">
|
|
||||||
</div>
|
|
||||||
|
|
||||||
10. تغيير اللغة والموضوع
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://github.com/binary-husky/gpt_academic/assets/96192199/b6799499-b6fb-4f0c-9c8e-1b441872f4e8" width="500" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
### II: الإصدارات:
|
|
||||||
- الإصدار 3.70 (قريبًا): تحسينات لوظائف AutoGen وتصميم سلسلة من المكونات المشتقة
|
|
||||||
- الإصدار 3.60: إدخال AutoGen كأساس لوظائف الجيل الجديد
|
|
||||||
- الإصدار 3.57: دعم GLM3، نار النجوم v3، وشجرة الكلمات v4، وإصلاح خطأ الازدحام في النماذج المحلية
|
|
||||||
- الإصدار 3.56: الدعم لإضافة مزامنة الأزرار الأساسية حسب الطلب، وصفحة تجميع تقارير البيانات في ملف PDF
|
|
||||||
- الإصدار 3.55: إعادة هيكلة واجهة المستخدم الأمامية، وإضافة نافذة عائمة وشريط قائمة
|
|
||||||
- الإصدار 3.54: إضافة مترجم الكود المباشر (Code Interpreter) (قيد الانجاز)
|
|
||||||
- الإصدار 3.53: دعم اختيار موضوعات واجهة مختلفة، وزيادة الاستقرار وحل مشاكل التعارض بين المستخدمين المتعدد
|
|
||||||
- الإصدار 3.50: استخدام اللغة الطبيعية لاستدعاء جميع وظائف المشروع هذا (محطة فراغ)، ودعم تصنيف الوظائف وتحسين واجهة المستخدم وتصميم مواضيع جديدة
|
|
||||||
- الإصدار 3.49: دعم المنصات البحثية في بيدو كونفان وشجرة الكلمات
|
|
||||||
- الإصدار 3.48: دعم علي بابا, بوكما رش حتكيا, إكسونامبلومانت النار
|
|
||||||
- الإصدار 3.46: دعم محادثة نصية في الوقت الحقيقي غير مراقبة
|
|
||||||
- الإصدار 3.45: دعم تخصيص LatexChatglm النموذج التعديل
|
|
||||||
- الإصدار 3.44: دعم Azure رسميًا، وتحسين سهولة الاستخدام للواجهات الأمامية
|
|
||||||
- الإصدار 3.4: +ترجمة النصوص الكاملة للمقالات من خلال ملف PDF، +اختيار موضع المنطقة النصية، +خيار التخطيط الرأسي، +تحسينات في وظائف التداخل العديدة
|
|
||||||
- الإصدار 3.3: +وظائف متكاملة للمعلومات عبر الإنترنت
|
|
||||||
- الإصدار 3.2: دعم وظائف المكونات التي تحتوي معلمات أكثر (حفظ النص، فهم أي لغة برمجة، طلب أي تركيبة LLM في وقت واحد)
|
|
||||||
- الإصدار 3.1: دعم السؤال نحو نماذج GPT المتعددة! دعم واجهة api2d، دعم توازن الأحمال بين المفاتيح الخاصة المتعددة
|
|
||||||
- الإصدار 3.0: دعم لنماذج جات، واحدة منها لشتلس الصغيرة
|
|
||||||
- الإصدار 2.6: إعادة تصميم بنية الوظائف، وتحسين التفاعل وإضافة مزيد من الوظائف
|
|
||||||
- الإصدار 2.5: التحديث التلقائي، وحل مشكلة النص الطويل عند ملخص المشاريع الضخمة وتجاوز النصوص.
|
|
||||||
- الإصدار 2.4: (١) إضافة ميزة ترجمة المقالات الدورية. (٢) إضافة ميزة لتحويل مكان منطقة الإدخال. (٣) إضافة خيار التخطيط العمودي (vertical layout). (٤) تحسين وظائف المكونات متعددة الخيوط.
|
|
||||||
- الإصدار 2.3: تحسين التفاعل مع مواضيع متعددة
|
|
||||||
- الإصدار 2.2: دعم إعادة تحميل الوظائف المكونة حراريًا
|
|
||||||
- الإصدار 2.1: تصميم قابل للطي
|
|
||||||
- الإصدار 2.0: إدخال وحدات الوظائف المكونة
|
|
||||||
- الإصدار 1.0: الوظائف الأساسية
|
|
||||||
|
|
||||||
مجموعة المطورين GPT Academic QQ: `610599535`
|
|
||||||
|
|
||||||
- مشكلات معروفة
|
|
||||||
- بعض ملحقات متصفح الترجمة تتداخل مع تشغيل الواجهة الأمامية لهذا البرنامج
|
|
||||||
- يحتوي Gradio الرسمي حاليًا على عدد كبير من مشاكل التوافق. يُرجى استخدام `requirement.txt` لتثبيت Gradio.
|
|
||||||
|
|
||||||
### III: الأنساق
|
|
||||||
يمكن تغيير الأنساق بتعديل خيار `THEME` (config.py)
|
|
||||||
1. `Chuanhu-Small-and-Beautiful` [الرابط](https://github.com/GaiZhenbiao/ChuanhuChatGPT/)
|
|
||||||
|
|
||||||
|
|
||||||
### IV: فروع تطوير هذا المشروع
|
|
||||||
|
|
||||||
1. الفرع `master`: الفرع الرئيسي، إصدار مستقر
|
|
||||||
2. الفرع `frontier`: الفرع التطويري، إصدار تجريبي
|
|
||||||
|
|
||||||
|
|
||||||
### V: المراجع والفروض التعليمية
|
|
||||||
|
|
||||||
```
|
|
||||||
استخدمت العديد من التصاميم الموجودة في مشاريع ممتازة أخرى في الأكواد التالية، للمراجع عشوائية:
|
|
||||||
|
|
||||||
# ViewGradio:
|
|
||||||
https://github.com/THUD
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
# مُثبّت بضغطة واحدة Oobabooga:
|
|
||||||
https://github.com/oobabooga/one-click-installers
|
|
||||||
|
|
||||||
# المزيد:
|
|
||||||
https://github.com/gradio-app/gradio
|
|
||||||
https://github.com/fghrsh/live2d_demo
|
|
||||||
@@ -1,357 +0,0 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
> **Note**
|
|
||||||
>
|
|
||||||
> This README was translated by GPT (implemented by the plugin of this project) and may not be 100% reliable. Please carefully check the translation results.
|
|
||||||
>
|
|
||||||
> 2023.11.7: When installing dependencies, please select the **specified versions** in the `requirements.txt` file. Installation command: `pip install -r requirements.txt`.
|
|
||||||
|
|
||||||
|
|
||||||
# <div align=center><img src="logo.png" width="40"> GPT Academic Optimization</div>
|
|
||||||
|
|
||||||
**If you like this project, please give it a Star.**
|
|
||||||
To translate this project to arbitrary language with GPT, read and run [`multi_language.py`](multi_language.py) (experimental).
|
|
||||||
|
|
||||||
> **Note**
|
|
||||||
>
|
|
||||||
> 1.Please note that only plugins (buttons) highlighted in **bold** support reading files, and some plugins are located in the **dropdown menu** in the plugin area. Additionally, we welcome and process any new plugins with the **highest priority** through PRs.
|
|
||||||
>
|
|
||||||
> 2.The functionalities of each file in this project are described in detail in the [self-analysis report `self_analysis.md`](https://github.com/binary-husky/gpt_academic/wiki/GPT‐Academic项目自译解报告). As the version iterates, you can also click on the relevant function plugin at any time to call GPT to regenerate the project's self-analysis report. Common questions are in the [`wiki`](https://github.com/binary-husky/gpt_academic/wiki). [Regular installation method](#installation) | [One-click installation script](https://github.com/binary-husky/gpt_academic/releases) | [Configuration instructions](https://github.com/binary-husky/gpt_academic/wiki/项目配置说明).
|
|
||||||
>
|
|
||||||
> 3.This project is compatible with and encourages the use of domestic large-scale language models such as ChatGLM. Multiple api-keys can be used together. You can fill in the configuration file with `API_KEY="openai-key1,openai-key2,azure-key3,api2d-key4"` to temporarily switch `API_KEY` during input, enter the temporary `API_KEY`, and then press enter to apply it.
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
<div align="center">
|
|
||||||
|
|
||||||
Feature (⭐ = Recently Added) | Description
|
|
||||||
--- | ---
|
|
||||||
⭐[Integrate New Models](https://github.com/binary-husky/gpt_academic/wiki/%E5%A6%82%E4%BD%95%E5%88%87%E6%8D%A2%E6%A8%A1%E5%9E%8B) | Baidu [Qianfan](https://cloud.baidu.com/doc/WENXINWORKSHOP/s/Nlks5zkzu) and Wenxin Yiyu, [Tongyi Qianwen](https://modelscope.cn/models/qwen/Qwen-7B-Chat/summary), Shanghai AI-Lab [Shusheng](https://github.com/InternLM/InternLM), Xunfei [Xinghuo](https://xinghuo.xfyun.cn/), [LLaMa2](https://huggingface.co/meta-llama/Llama-2-7b-chat-hf), Zhifu API, DALLE3
|
|
||||||
Proofreading, Translation, Code Explanation | One-click proofreading, translation, searching for grammar errors in papers, explaining code
|
|
||||||
[Custom Shortcuts](https://www.bilibili.com/video/BV14s4y1E7jN) | Support for custom shortcuts
|
|
||||||
Modular Design | Support for powerful [plugins](https://github.com/binary-husky/gpt_academic/tree/master/crazy_functions), plugins support [hot updates](https://github.com/binary-husky/gpt_academic/wiki/%E5%87%BD%E6%95%B0%E6%8F%92%E4%BB%B6%E6%8C%87%E5%8D%97)
|
|
||||||
[Program Profiling](https://www.bilibili.com/video/BV1cj411A7VW) | [Plugin] One-click to profile Python/C/C++/Java/Lua/... project trees or [self-profiling](https://www.bilibili.com/video/BV1cj411A7VW)
|
|
||||||
Read Papers, [Translate](https://www.bilibili.com/video/BV1KT411x7Wn) Papers | [Plugin] One-click to interpret full-text latex/pdf papers and generate abstracts
|
|
||||||
Full-text Latex [Translation](https://www.bilibili.com/video/BV1nk4y1Y7Js/), [Proofreading](https://www.bilibili.com/video/BV1FT411H7c5/) | [Plugin] One-click translation or proofreading of latex papers
|
|
||||||
Batch Comment Generation | [Plugin] One-click batch generation of function comments
|
|
||||||
Markdown [Translation](https://www.bilibili.com/video/BV1yo4y157jV/) | [Plugin] Did you see the [README](https://github.com/binary-husky/gpt_academic/blob/master/docs/README_EN.md) in the top five languages?
|
|
||||||
Chat Analysis Report Generation | [Plugin] Automatically generates summary reports after running
|
|
||||||
[PDF Paper Full-text Translation](https://www.bilibili.com/video/BV1KT411x7Wn) | [Plugin] Extract title & abstract of PDF papers + translate full-text (multi-threaded)
|
|
||||||
[Arxiv Helper](https://www.bilibili.com/video/BV1LM4y1279X) | [Plugin] Enter the arxiv article URL to translate the abstract + download PDF with one click
|
|
||||||
One-click Proofreading of Latex Papers | [Plugin] Syntax and spelling correction of Latex papers similar to Grammarly + output side-by-side PDF
|
|
||||||
[Google Scholar Integration Helper](https://www.bilibili.com/video/BV19L411U7ia) | [Plugin] Given any Google Scholar search page URL, let GPT help you [write related works](https://www.bilibili.com/video/BV1GP411U7Az/)
|
|
||||||
Internet Information Aggregation + GPT | [Plugin] One-click to let GPT retrieve information from the Internet to answer questions and keep the information up to date
|
|
||||||
⭐Arxiv Paper Fine Translation ([Docker](https://github.com/binary-husky/gpt_academic/pkgs/container/gpt_academic_with_latex)) | [Plugin] One-click [high-quality translation of arxiv papers](https://www.bilibili.com/video/BV1dz4y1v77A/), the best paper translation tool at present
|
|
||||||
⭐[Real-time Speech Input](https://github.com/binary-husky/gpt_academic/blob/master/docs/use_audio.md) | [Plugin] Asynchronously [listen to audio](https://www.bilibili.com/video/BV1AV4y187Uy/), automatically segment sentences, and automatically find the best time to answer
|
|
||||||
Formula/Image/Table Display | Can simultaneously display formulas in [TeX form and rendered form](https://user-images.githubusercontent.com/96192199/230598842-1d7fcddd-815d-40ee-af60-baf488a199df.png), support formula and code highlighting
|
|
||||||
⭐AutoGen Multi-Agent Plugin | [Plugin] Explore the emergence of multi-agent intelligence with Microsoft AutoGen!
|
|
||||||
Start Dark [Theme](https://github.com/binary-husky/gpt_academic/issues/173) | Add ```/?__theme=dark``` to the end of the browser URL to switch to the dark theme
|
|
||||||
[More LLM Model Support](https://www.bilibili.com/video/BV1wT411p7yf) | It must be great to be served by GPT3.5, GPT4, [THU ChatGLM2](https://github.com/THUDM/ChatGLM2-6B), and [Fudan MOSS](https://github.com/OpenLMLab/MOSS) at the same time, right?
|
|
||||||
⭐ChatGLM2 Fine-tuning Model | Support for loading ChatGLM2 fine-tuning models and providing ChatGLM2 fine-tuning assistant plugins
|
|
||||||
More LLM Model Access, support for [huggingface deployment](https://huggingface.co/spaces/qingxu98/gpt-academic) | Join NewBing interface (New Bing), introduce Tsinghua [JittorLLMs](https://github.com/Jittor/JittorLLMs) to support [LLaMA](https://github.com/facebookresearch/llama) and [Pangu](https://openi.org.cn/pangu/)
|
|
||||||
⭐[void-terminal](https://github.com/binary-husky/void-terminal) pip package | Use this project's all function plugins directly in Python without GUI (under development)
|
|
||||||
⭐Void Terminal Plugin | [Plugin] Schedule other plugins of this project directly in natural language
|
|
||||||
More New Feature Demonstrations (Image Generation, etc.)...... | See the end of this document ........
|
|
||||||
</div>
|
|
||||||
|
|
||||||
|
|
||||||
- New interface (modify the LAYOUT option in `config.py` to switch between "left-right layout" and "top-bottom layout")
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://github.com/binary-husky/gpt_academic/assets/96192199/d81137c3-affd-4cd1-bb5e-b15610389762" width="700" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
|
|
||||||
- All buttons are dynamically generated by reading `functional.py` and can be added with custom functions to free up the clipboard
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/231975334-b4788e91-4887-412f-8b43-2b9c5f41d248.gif" width="700" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
- Proofreading/Correction
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/231980294-f374bdcb-3309-4560-b424-38ef39f04ebd.gif" width="700" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
- If the output contains formulas, they will be displayed in both tex format and rendered format for easy copying and reading.
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/230598842-1d7fcddd-815d-40ee-af60-baf488a199df.png" width="700" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
- Too lazy to look at the project code? Show off the whole project directly in chatgpt's mouth
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/226935232-6b6a73ce-8900-4aee-93f9-733c7e6fef53.png" width="700" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
- Multiple large language models mixed calling (ChatGLM + OpenAI-GPT3.5 + [API2D](https://api2d.com/)-GPT4)
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/232537274-deca0563-7aa6-4b5d-94a2-b7c453c47794.png" width="700" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
# Installation
|
|
||||||
### Installation Method I: Run directly (Windows, Linux or MacOS)
|
|
||||||
|
|
||||||
1. Download the project
|
|
||||||
```sh
|
|
||||||
git clone --depth=1 https://github.com/binary-husky/gpt_academic.git
|
|
||||||
cd gpt_academic
|
|
||||||
```
|
|
||||||
|
|
||||||
2. Configure API_KEY
|
|
||||||
|
|
||||||
In `config.py`, configure API KEY and other settings, [click here to see special network environment configuration methods](https://github.com/binary-husky/gpt_academic/issues/1). [Wiki page](https://github.com/binary-husky/gpt_academic/wiki/项目配置说明)。
|
|
||||||
|
|
||||||
「 The program will first check if a secret configuration file named `config_private.py` exists and use the configurations from that file to override the ones in `config.py` with the same names. If you understand this logic, we strongly recommend that you create a new configuration file named `config_private.py` next to `config.py` and move (copy) the configurations from `config.py` to `config_private.py` (only copy the configuration items you have modified). 」
|
|
||||||
|
|
||||||
「 Project configuration can be done via `environment variables`. The format of the environment variables can be found in the `docker-compose.yml` file or our [Wiki page](https://github.com/binary-husky/gpt_academic/wiki/项目配置说明). Configuration priority: `environment variables` > `config_private.py` > `config.py`. 」
|
|
||||||
|
|
||||||
|
|
||||||
3. Install dependencies
|
|
||||||
```sh
|
|
||||||
# (Option I: If you are familiar with python, python>=3.9) Note: Use the official pip source or the Aliyun pip source. Temporary method for switching the source: python -m pip install -r requirements.txt -i https://mirrors.aliyun.com/pypi/simple/
|
|
||||||
python -m pip install -r requirements.txt
|
|
||||||
|
|
||||||
# (Option II: Using Anaconda) The steps are similar (https://www.bilibili.com/video/BV1rc411W7Dr):
|
|
||||||
conda create -n gptac_venv python=3.11 # Create the anaconda environment
|
|
||||||
conda activate gptac_venv # Activate the anaconda environment
|
|
||||||
python -m pip install -r requirements.txt # This step is the same as the pip installation process
|
|
||||||
```
|
|
||||||
|
|
||||||
|
|
||||||
<details><summary>If you need to support THU ChatGLM2, Fudan MOSS, or RWKV Runner as backends, click here to expand</summary>
|
|
||||||
<p>
|
|
||||||
|
|
||||||
【Optional Step】If you need to support THU ChatGLM2 or Fudan MOSS as backends, you need to install additional dependencies (Prerequisites: Familiar with Python + Familiar with Pytorch + Sufficient computer configuration):
|
|
||||||
```sh
|
|
||||||
# 【Optional Step I】Support THU ChatGLM2. Note: If you encounter the "Call ChatGLM fail unable to load ChatGLM parameters" error, refer to the following: 1. The default installation above is for torch+cpu version. To use cuda, uninstall torch and reinstall torch+cuda; 2. If the model cannot be loaded due to insufficient local configuration, you can modify the model accuracy in request_llm/bridge_chatglm.py. Change AutoTokenizer.from_pretrained("THUDM/chatglm-6b", trust_remote_code=True) to AutoTokenizer.from_pretrained("THUDM/chatglm-6b-int4", trust_remote_code=True)
|
|
||||||
python -m pip install -r request_llms/requirements_chatglm.txt
|
|
||||||
|
|
||||||
# 【Optional Step II】Support Fudan MOSS
|
|
||||||
python -m pip install -r request_llms/requirements_moss.txt
|
|
||||||
git clone --depth=1 https://github.com/OpenLMLab/MOSS.git request_llms/moss # When executing this line of code, make sure you are in the root directory of the project
|
|
||||||
|
|
||||||
# 【Optional Step III】Support RWKV Runner
|
|
||||||
Refer to wiki: https://github.com/binary-husky/gpt_academic/wiki/%E9%80%82%E9%85%8DRWKV-Runner
|
|
||||||
|
|
||||||
# 【Optional Step IV】Make sure that the AVAIL_LLM_MODELS in the config.py configuration file includes the expected models. The currently supported models are as follows (jittorllms series currently only supports the docker solution):
|
|
||||||
AVAIL_LLM_MODELS = ["gpt-3.5-turbo", "api2d-gpt-3.5-turbo", "gpt-4", "api2d-gpt-4", "chatglm", "moss"] # + ["jittorllms_rwkv", "jittorllms_pangualpha", "jittorllms_llama"]
|
|
||||||
```
|
|
||||||
|
|
||||||
</p>
|
|
||||||
</details>
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
4. Run
|
|
||||||
```sh
|
|
||||||
python main.py
|
|
||||||
```
|
|
||||||
|
|
||||||
### Installation Method II: Use Docker
|
|
||||||
|
|
||||||
0. Deploy all capabilities of the project (this is a large image that includes cuda and latex. Not recommended if you have slow internet speed or small hard drive)
|
|
||||||
[](https://github.com/binary-husky/gpt_academic/actions/workflows/build-with-all-capacity.yml)
|
|
||||||
|
|
||||||
``` sh
|
|
||||||
# Modify docker-compose.yml, keep scheme 0 and delete other schemes. Then run:
|
|
||||||
docker-compose up
|
|
||||||
```
|
|
||||||
|
|
||||||
1. ChatGPT + Wenxin + Spark online models only (recommended for most people)
|
|
||||||
[](https://github.com/binary-husky/gpt_academic/actions/workflows/build-without-local-llms.yml)
|
|
||||||
[](https://github.com/binary-husky/gpt_academic/actions/workflows/build-with-latex.yml)
|
|
||||||
[](https://github.com/binary-husky/gpt_academic/actions/workflows/build-with-audio-assistant.yml)
|
|
||||||
|
|
||||||
``` sh
|
|
||||||
# Modify docker-compose.yml, keep scheme 1 and delete other schemes. Then run:
|
|
||||||
docker-compose up
|
|
||||||
```
|
|
||||||
|
|
||||||
P.S. If you need the latex plugin functionality, please see the Wiki. Also, you can directly use scheme 4 or scheme 0 to get the Latex functionality.
|
|
||||||
|
|
||||||
2. ChatGPT + ChatGLM2 + MOSS + LLAMA2 + Intelligent Questions (requires familiarity with [Nvidia Docker](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html#installing-on-ubuntu-and-debian) runtime)
|
|
||||||
[](https://github.com/binary-husky/gpt_academic/actions/workflows/build-with-chatglm.yml)
|
|
||||||
|
|
||||||
``` sh
|
|
||||||
# Modify docker-compose.yml, keep scheme 2 and delete other schemes. Then run:
|
|
||||||
docker-compose up
|
|
||||||
```
|
|
||||||
|
|
||||||
|
|
||||||
### Installation Method III: Other deployment methods
|
|
||||||
1. **Windows one-click running script**.
|
|
||||||
Windows users who are completely unfamiliar with the python environment can download the one-click running script from the [Release](https://github.com/binary-husky/gpt_academic/releases) to install the version without local models.
|
|
||||||
The script is contributed by [oobabooga](https://github.com/oobabooga/one-click-installers).
|
|
||||||
|
|
||||||
2. Use third-party APIs, Azure, Wenxin, Xinghuo, etc., see [Wiki page](https://github.com/binary-husky/gpt_academic/wiki/项目配置说明)
|
|
||||||
|
|
||||||
3. Pitfall guide for deploying on cloud servers.
|
|
||||||
Please visit [Cloud Server Remote Deployment Wiki](https://github.com/binary-husky/gpt_academic/wiki/%E4%BA%91%E6%9C%8D%E5%8A%A1%E5%99%A8%E8%BF%9C%E7%A8%8B%E9%83%A8%E7%BD%B2%E6%8C%87%E5%8D%97)
|
|
||||||
|
|
||||||
4. Some new deployment platforms or methods
|
|
||||||
- Use Sealos [to deploy with one click](https://github.com/binary-husky/gpt_academic/issues/993).
|
|
||||||
- Use WSL2 (Windows Subsystem for Linux). Please refer to [Deployment Wiki-2](https://github.com/binary-husky/gpt_academic/wiki/%E4%BD%BF%E7%94%A8WSL2%EF%BC%88Windows-Subsystem-for-Linux-%E5%AD%90%E7%B3%BB%E7%BB%9F%EF%BC%89%E9%83%A8%E7%BD%B2)
|
|
||||||
- How to run under a subpath (such as `http://localhost/subpath`). Please visit [FastAPI Run Instructions](docs/WithFastapi.md)
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
# Advanced Usage
|
|
||||||
### I: Customizing new convenient buttons (academic shortcuts)
|
|
||||||
Open `core_functional.py` with any text editor, add the following entry, and then restart the program. (If the button already exists, both the prefix and suffix can be modified on-the-fly without restarting the program.)
|
|
||||||
For example:
|
|
||||||
```
|
|
||||||
"Super Translation": {
|
|
||||||
# Prefix: will be added before your input. For example, used to describe your request, such as translation, code explanation, proofreading, etc.
|
|
||||||
"Prefix": "Please translate the following paragraph into Chinese and then explain each proprietary term in the text using a markdown table:\n\n",
|
|
||||||
|
|
||||||
# Suffix: will be added after your input. For example, used to wrap your input in quotation marks along with the prefix.
|
|
||||||
"Suffix": "",
|
|
||||||
},
|
|
||||||
```
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/226899272-477c2134-ed71-4326-810c-29891fe4a508.png" width="500" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
### II: Custom function plugins
|
|
||||||
Write powerful function plugins to perform any task you desire and can't imagine.
|
|
||||||
The difficulty of writing and debugging plugins in this project is very low. As long as you have a certain knowledge of Python, you can implement your own plugin functionality by following the template we provide.
|
|
||||||
For more details, please refer to the [Function Plugin Guide](https://github.com/binary-husky/gpt_academic/wiki/%E5%87%BD%E6%95%B0%E6%8F%92%E4%BB%B6%E6%8C%87%E5%8D%97).
|
|
||||||
|
|
||||||
# Updates
|
|
||||||
### I: Dynamics
|
|
||||||
|
|
||||||
1. Conversation-saving feature. Call `Save the current conversation` in the function plugin area to save the current conversation as a readable and restorable HTML file. Additionally, call `Load conversation history archive` in the function plugin area (drop-down menu) to restore previous sessions.
|
|
||||||
Tip: Clicking `Load conversation history archive` without specifying a file allows you to view the cached historical HTML archive.
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/235222390-24a9acc0-680f-49f5-bc81-2f3161f1e049.png" width="500" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
2. ⭐Latex/Arxiv paper translation feature⭐
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://github.com/binary-husky/gpt_academic/assets/96192199/002a1a75-ace0-4e6a-94e2-ec1406a746f1" height="250" > ===>
|
|
||||||
<img src="https://github.com/binary-husky/gpt_academic/assets/96192199/9fdcc391-f823-464f-9322-f8719677043b" height="250" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
3. Void Terminal (understanding user intent from natural language input and automatically calling other plugins)
|
|
||||||
|
|
||||||
- Step 1: Enter " Please call the plugin to translate the PDF paper, the address is https://openreview.net/pdf?id=rJl0r3R9KX"
|
|
||||||
- Step 2: Click "Void Terminal"
|
|
||||||
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://github.com/binary-husky/gpt_academic/assets/96192199/66f1b044-e9ff-4eed-9126-5d4f3668f1ed" width="500" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
4. Modular function design, simple interface supporting powerful functionality
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/229288270-093643c1-0018-487a-81e6-1d7809b6e90f.png" height="400" >
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/227504931-19955f78-45cd-4d1c-adac-e71e50957915.png" height="400" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
5. Translate and interpret other open-source projects
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/226935232-6b6a73ce-8900-4aee-93f9-733c7e6fef53.png" height="250" >
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/226969067-968a27c1-1b9c-486b-8b81-ab2de8d3f88a.png" height="250" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
6. Added small features that decorate [live2d](https://github.com/fghrsh/live2d_demo) (disabled by default, needs modification in `config.py`)
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/236432361-67739153-73e8-43fe-8111-b61296edabd9.png" width="500" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
7. OpenAI image generation
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://github.com/binary-husky/gpt_academic/assets/96192199/bc7ab234-ad90-48a0-8d62-f703d9e74665" width="500" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
8. OpenAI audio parsing and summarization
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://github.com/binary-husky/gpt_academic/assets/96192199/709ccf95-3aee-498a-934a-e1c22d3d5d5b" width="500" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
9. Latex full-text proofreading and correction
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://github.com/binary-husky/gpt_academic/assets/96192199/651ccd98-02c9-4464-91e1-77a6b7d1b033" height="200" > ===>
|
|
||||||
<img src="https://github.com/binary-husky/gpt_academic/assets/96192199/476f66d9-7716-4537-b5c1-735372c25adb" height="200">
|
|
||||||
</div>
|
|
||||||
|
|
||||||
10. Language and theme switching
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://github.com/binary-husky/gpt_academic/assets/96192199/b6799499-b6fb-4f0c-9c8e-1b441872f4e8" width="500" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
### II: Versions:
|
|
||||||
- version 3.70 (todo): Optimize the AutoGen plugin theme and design a series of derivative plugins
|
|
||||||
- version 3.60: Introduce AutoGen as the cornerstone of the new generation of plugins
|
|
||||||
- version 3.57: Support GLM3, Spark v3, Wenxin Quote v4, and fix concurrency bugs in local models
|
|
||||||
- version 3.56: Support dynamically adding basic functional buttons and a new summary PDF page
|
|
||||||
- version 3.55: Refactor the frontend interface and introduce floating windows and a menu bar
|
|
||||||
- version 3.54: Add a dynamic code interpreter (Code Interpreter) (to be improved)
|
|
||||||
- version 3.53: Support dynamically choosing different interface themes, improve stability, and resolve conflicts between multiple users
|
|
||||||
- version 3.50: Use natural language to call all function plugins of this project (Void Terminal), support plugin classification, improve UI, and design new themes
|
|
||||||
- version 3.49: Support Baidu Qianfan Platform and Wenxin Quote
|
|
||||||
- version 3.48: Support Ali Dharma Academy Tongyi Qianwen, Shanghai AI-Lab Shusheng, and Xunfei Spark
|
|
||||||
- version 3.46: Support fully hands-off real-time voice conversation
|
|
||||||
- version 3.45: Support customizing ChatGLM2 fine-tuned models
|
|
||||||
- version 3.44: Officially support Azure, optimize interface usability
|
|
||||||
- version 3.4: + Arxiv paper translation, latex paper correction functionality
|
|
||||||
- version 3.3: + Internet information integration functionality
|
|
||||||
- version 3.2: Function plugins support more parameter interfaces (conversation saving functionality, interpreting any code language + asking any combination of LLMs simultaneously)
|
|
||||||
- version 3.1: Support querying multiple GPT models simultaneously! Support API2D, support load balancing for multiple API keys
|
|
||||||
- version 3.0: Support chatglm and other small-scale LLMs
|
|
||||||
- version 2.6: Refactored plugin structure, improved interactivity, added more plugins
|
|
||||||
- version 2.5: Self-updating, fix the problem of text being too long and token overflowing when summarizing large code projects
|
|
||||||
- version 2.4: (1) Add PDF full-text translation functionality; (2) Add functionality to switch the position of the input area; (3) Add vertical layout option; (4) Optimize multi-threaded function plugins.
|
|
||||||
- version 2.3: Enhance multi-threaded interactivity
|
|
||||||
- version 2.2: Function plugin hot-reloading support
|
|
||||||
- version 2.1: Collapsible layout
|
|
||||||
- version 2.0: Introduce modular function plugins
|
|
||||||
- version 1.0: Basic functionality
|
|
||||||
|
|
||||||
GPT Academic Developer QQ Group: `610599535`
|
|
||||||
|
|
||||||
- Known Issues
|
|
||||||
- Some browser translation plugins interfere with the frontend operation of this software
|
|
||||||
- Official Gradio currently has many compatibility bugs, please make sure to install Gradio using `requirement.txt`
|
|
||||||
|
|
||||||
### III: Themes
|
|
||||||
You can change the theme by modifying the `THEME` option (config.py).
|
|
||||||
1. `Chuanhu-Small-and-Beautiful` [Website](https://github.com/GaiZhenbiao/ChuanhuChatGPT/)
|
|
||||||
|
|
||||||
### IV: Development Branches of This Project
|
|
||||||
|
|
||||||
1. `master` branch: Main branch, stable version
|
|
||||||
2. `frontier` branch: Development branch, test version
|
|
||||||
|
|
||||||
***
|
|
||||||
|
|
||||||
### V: References and Learning
|
|
||||||
|
|
||||||
|
|
||||||
The code references the designs of many other excellent projects, in no particular order:
|
|
||||||
|
|
||||||
[THU ChatGLM2-6B](https://github.com/THUDM/ChatGLM2-6B)
|
|
||||||
|
|
||||||
|
|
||||||
[THU JittorLLMs](https://github.com/Jittor/JittorLLMs)
|
|
||||||
|
|
||||||
|
|
||||||
[ChatPaper](https://github.com/kaixindelele/ChatPaper)
|
|
||||||
|
|
||||||
|
|
||||||
[Edge-GPT](https://github.com/acheong08/EdgeGPT)
|
|
||||||
|
|
||||||
|
|
||||||
[ChuanhuChatGPT](https://github.com/GaiZhenbiao/ChuanhuChatGPT)
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
# Oobabooga one-click installer:
|
|
||||||
https://github.com/oobabooga/one-click-installers
|
|
||||||
|
|
||||||
# More:
|
|
||||||
https://github.com/gradio-app/gradio
|
|
||||||
https://github.com/fghrsh/live2d_demo
|
|
||||||
@@ -1,356 +0,0 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
> **Remarque**
|
|
||||||
>
|
|
||||||
> Ce README a été traduit par GPT (implémenté par le plugin de ce projet) et n'est pas fiable à 100 %. Veuillez examiner attentivement les résultats de la traduction.
|
|
||||||
>
|
|
||||||
> 7 novembre 2023 : Lors de l'installation des dépendances, veuillez choisir les versions **spécifiées** dans le fichier `requirements.txt`. Commande d'installation : `pip install -r requirements.txt`.
|
|
||||||
|
|
||||||
|
|
||||||
# <div align=center><img src="logo.png" width="40"> Optimisation académique GPT (GPT Academic)</div>
|
|
||||||
|
|
||||||
**Si vous aimez ce projet, merci de lui donner une étoile ; si vous avez inventé des raccourcis ou des plugins utiles, n'hésitez pas à envoyer des demandes d'extraction !**
|
|
||||||
|
|
||||||
Si vous aimez ce projet, veuillez lui donner une étoile.
|
|
||||||
Pour traduire ce projet dans une langue arbitraire avec GPT, lisez et exécutez [`multi_language.py`](multi_language.py) (expérimental).
|
|
||||||
|
|
||||||
> **Remarque**
|
|
||||||
>
|
|
||||||
> 1. Veuillez noter que seuls les plugins (boutons) marqués en **surbrillance** prennent en charge la lecture de fichiers, et certains plugins se trouvent dans le **menu déroulant** de la zone des plugins. De plus, nous accueillons avec la plus haute priorité les nouvelles demandes d'extraction de plugins.
|
|
||||||
>
|
|
||||||
> 2. Les fonctionnalités de chaque fichier de ce projet sont spécifiées en détail dans [le rapport d'auto-analyse `self_analysis.md`](https://github.com/binary-husky/gpt_academic/wiki/GPT‐Academic个项目自译解报告). Vous pouvez également cliquer à tout moment sur les plugins de fonctions correspondants pour appeler GPT et générer un rapport d'auto-analyse du projet. Questions fréquemment posées [wiki](https://github.com/binary-husky/gpt_academic/wiki). [Méthode d'installation standard](#installation) | [Script d'installation en un clic](https://github.com/binary-husky/gpt_academic/releases) | [Instructions de configuration](https://github.com/binary-husky/gpt_academic/wiki/项目配置说明)..
|
|
||||||
>
|
|
||||||
> 3. Ce projet est compatible avec et recommande l'expérimentation de grands modèles de langage chinois tels que ChatGLM, etc. Prend en charge plusieurs clés API, vous pouvez les remplir dans le fichier de configuration comme `API_KEY="openai-key1,openai-key2,azure-key3,api2d-key4"`. Pour changer temporairement la clé API, entrez la clé API temporaire dans la zone de saisie, puis appuyez sur Entrée pour soumettre et activer celle-ci.
|
|
||||||
|
|
||||||
|
|
||||||
<div align="center">
|
|
||||||
|
|
||||||
Fonctionnalités (⭐ = fonctionnalité récemment ajoutée) | Description
|
|
||||||
--- | ---
|
|
||||||
⭐[Modèles acquis](https://github.com/binary-husky/gpt_academic/wiki/如何切换模型)！ | Baidu [Qianfan](https://cloud.baidu.com/doc/WENXINWORKSHOP/s/Nlks5zkzu) et Wenxin Yiyuan, [Tongyi Qianwen](https://modelscope.cn/models/qwen/Qwen-7B-Chat/summary), Shanghai AI-Lab [Shusheng](https://github.com/InternLM/InternLM), Xunfei [Xinghuo](https://xinghuo.xfyun.cn/), [LLaMa2](https://huggingface.co/meta-llama/Llama-2-7b-chat-hf), Zhifu API, DALLE3
|
|
||||||
Amélioration, traduction, explication du code | Correction, traduction, recherche d'erreurs de syntaxe dans les articles, explication du code
|
|
||||||
[Raccourcis personnalisés](https://www.bilibili.com/video/BV14s4y1E7jN) | Prise en charge de raccourcis personnalisés
|
|
||||||
Conception modulaire | Prise en charge de plugins puissants personnalisables, prise en charge de la [mise à jour à chaud](https://github.com/binary-husky/gpt_academic/wiki/函数插件指南) des plugins
|
|
||||||
[Analyse de programme](https://www.bilibili.com/video/BV1cj411A7VW) | [Plugin] Analyse en profondeur d'un arbre de projets Python/C/C++/Java/Lua/... d'un simple clic ou [auto-analyse](https://www.bilibili.com/video/BV1cj411A7VW)
|
|
||||||
Lecture d'articles, traduction d'articles | [Plugin] Lecture automatique des articles LaTeX/PDF et génération du résumé
|
|
||||||
Traduction complète de [LaTeX](https://www.bilibili.com/video/BV1nk4y1Y7Js/) ou amélioration de leur qualité | [Plugin] Traduction ou amélioration rapide des articles LaTeX
|
|
||||||
Génération de commentaires en masse | [Plugin] Génération facile de commentaires de fonctions
|
|
||||||
Traduction [chinois-anglais](https://www.bilibili.com/video/BV1yo4y157jV/) du Markdown | [Plugin] Avez-vous vu le [README](https://github.com/binary-husky/gpt_academic/blob/master/docs/README_EN.md) dans les cinq langues ci-dessus ?
|
|
||||||
Génération de rapports d'analyse du chat | [Plugin] Génération automatique d'un rapport récapitulatif après l'exécution du chat
|
|
||||||
[Fonction de traduction complète des articles PDF](https://www.bilibili.com/video/BV1KT411x7Wn) | [Plugin] Extraction du titre et du résumé d'un article PDF, ainsi que traduction intégrale (multithreading)
|
|
||||||
Assistant Arxiv | [Plugin] Saisissez l'URL d'un article Arxiv pour traduire automatiquement le résumé et télécharger le PDF
|
|
||||||
Correction automatique d'articles LaTeX | [Plugin] Correction de la grammaire, de l'orthographe et comparaison avec le PDF correspondant, à la manière de Grammarly
|
|
||||||
Assistant Google Scholar | [Plugin] Donner l'URL d'une page de recherche Google Scholar pour obtenir de l'aide sur l'écriture des références
|
|
||||||
Agrégation d'informations sur Internet + GPT | [Plugin] Obtenez les informations de l'Internet pour répondre aux questions à l'aide de GPT, afin que les informations ne soient jamais obsolètes
|
|
||||||
⭐Traduction détaillée des articles Arxiv ([Docker](https://github.com/binary-husky/gpt_academic/pkgs/container/gpt_academic_with_latex)) | [Plugin] Traduction de haute qualité d'articles Arxiv en un clic, le meilleur outil de traduction d'articles à ce jour
|
|
||||||
⭐[Saisie orale en temps réel](https://github.com/binary-husky/gpt_academic/blob/master/docs/use_audio.md) | [Plugin] Écoute asynchrone de l'audio, découpage automatique et recherche automatique du meilleur moment pour répondre
|
|
||||||
Affichage des formules, images, tableaux | Affichage simultané de la forme [TeX et rendue](https://user-images.githubusercontent.com/96192199/230598842-1d7fcddd-815d-40ee-af60-baf488a199df.png) des formules, prise en charge de la mise en évidence des formules et du code
|
|
||||||
⭐Plugin AutoGen multi-agents | [Plugin] Explorez les émergences intelligentes à plusieurs agents avec Microsoft AutoGen !
|
|
||||||
Activation du [thème sombre](https://github.com/binary-husky/gpt_academic/issues/173) | Ajouter ```/?__theme=dark``` à l'URL du navigateur pour basculer vers le thème sombre
|
|
||||||
Prise en charge de plusieurs modèles LLM | Expérimentez avec GPT 3.5, GPT4, [ChatGLM2 de Tsinghua](https://github.com/THUDM/ChatGLM2-6B), [MOSS de Fudan](https://github.com/OpenLMLab/MOSS) simultanément !
|
|
||||||
⭐Modèle ChatGLM2 fine-tuned | Chargez et utilisez un modèle fine-tuned de ChatGLM2, disponible avec un plugin d'assistance
|
|
||||||
Prise en charge de plus de modèles LLM, déploiement sur [Huggingface](https://huggingface.co/spaces/qingxu98/gpt-academic) | Ajout de l'interface de connaissance-API, support de [LLaMA](https://github.com/facebookresearch/llama) et [PanGuα](https://openi.org.cn/pangu/)
|
|
||||||
⭐Paquet pip [void-terminal](https://github.com/binary-husky/void-terminal) | Accédez à toutes les fonctions et plugins de ce projet directement depuis Python (en cours de développement)
|
|
||||||
⭐Plugin terminal du vide | [Plugin] Utilisez un langage naturel pour interagir avec les autres plugins du projet
|
|
||||||
Affichage de nouvelles fonctionnalités (génération d'images, etc.) …… | Voir à la fin de ce document ……
|
|
||||||
</div>
|
|
||||||
|
|
||||||
|
|
||||||
- Nouvelle interface (modifiez l'option LAYOUT dans `config.py` pour basculer entre la disposition "gauche-droite" et "haut-bas")
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://github.com/binary-husky/gpt_academic/assets/96192199/d81137c3-affd-4cd1-bb5e-b15610389762" width="700" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
|
|
||||||
- Tous les boutons sont générés dynamiquement en lisant `functional.py`, vous pouvez donc ajouter de nouvelles fonctionnalités personnalisées et libérer le presse-papiers.
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/231975334-b4788e91-4887-412f-8b43-2b9c5f41d248.gif" width="700" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
- Retouche/correction
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/231980294-f374bdcb-3309-4560-b424-38ef39f04ebd.gif" width="700" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
- If the output contains formulas, they will be displayed in both tex and rendered forms for easy copying and reading.
|
|
||||||
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/230598842-1d7fcddd-815d-40ee-af60-baf488a199df.png" width="700" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
- Don't feel like looking at the project code? Just give it to ChatGPT to show off.
|
|
||||||
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/226935232-6b6a73ce-8900-4aee-93f9-733c7e6fef53.png" width="700" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
- Multiple large language models are mixed and used together (ChatGLM + OpenAI-GPT3.5 + [API2D](https://api2d.com/)-GPT4).
|
|
||||||
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/232537274-deca0563-7aa6-4b5d-94a2-b7c453c47794.png" width="700" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
# Installation
|
|
||||||
### Method I: Run directly (Windows, Linux, or MacOS)
|
|
||||||
|
|
||||||
1. Download the project
|
|
||||||
```sh
|
|
||||||
git clone --depth=1 https://github.com/binary-husky/gpt_academic.git
|
|
||||||
cd gpt_academic
|
|
||||||
```
|
|
||||||
|
|
||||||
2. Configure API_KEY
|
|
||||||
|
|
||||||
In `config.py`, configure the API KEY and other settings. [Click here to see methods for special network environment configurations](https://github.com/binary-husky/gpt_academic/issues/1). [Wiki page](https://github.com/binary-husky/gpt_academic/wiki/项目配置说明).
|
|
||||||
|
|
||||||
「 The program will first check if there is a confidential configuration file named `config_private.py`, and use the configurations in that file to override the corresponding configurations in `config.py`. If you understand this logic, we strongly recommend creating a new configuration file named `config_private.py` right next to `config.py`, and move (copy) the configurations from `config.py` to `config_private.py` (only copy the configurations that you have modified). 」
|
|
||||||
|
|
||||||
「 You can also configure the project using `environment variables`. The format of the environment variables can be found in the `docker-compose.yml` file or on our [Wiki page](https://github.com/binary-husky/gpt_academic/wiki/项目配置说明). The priority of configuration reading is: `environment variables` > `config_private.py` > `config.py`. 」
|
|
||||||
|
|
||||||
3. Install dependencies
|
|
||||||
```sh
|
|
||||||
# (Option I: If you are familiar with Python, python>=3.9) Note: Use the official pip source or the Ali pip source. Temporary change of source method: python -m pip install -r requirements.txt -i https://mirrors.aliyun.com/pypi/simple/
|
|
||||||
python -m pip install -r requirements.txt
|
|
||||||
|
|
||||||
# (Option II: Use Anaconda) The steps are similar (https://www.bilibili.com/video/BV1rc411W7Dr):
|
|
||||||
conda create -n gptac_venv python=3.11 # Create an anaconda environment
|
|
||||||
conda activate gptac_venv # Activate the anaconda environment
|
|
||||||
python -m pip install -r requirements.txt # This step is the same as the pip installation step
|
|
||||||
```
|
|
||||||
|
|
||||||
|
|
||||||
<details><summary>If you need to support Tsinghua ChatGLM2/Fudan MOSS/RWKV as backends, click here to expand</summary>
|
|
||||||
<p>
|
|
||||||
|
|
||||||
[Optional Steps] If you need to support Tsinghua ChatGLM2/Fudan MOSS as backends, you need to install additional dependencies (Prerequisites: Familiar with Python + Have used PyTorch + Sufficient computer configuration):
|
|
||||||
```sh
|
|
||||||
# [Optional Step I] Support Tsinghua ChatGLM2. Comment on this note: If you encounter the error "Call ChatGLM generated an error and cannot load the parameters of ChatGLM", refer to the following: 1: The default installation is the torch+cpu version. To use cuda, you need to uninstall torch and reinstall torch+cuda; 2: If the model cannot be loaded due to insufficient computer configuration, you can modify the model precision in request_llm/bridge_chatglm.py. Change AutoTokenizer.from_pretrained("THUDM/chatglm-6b", trust_remote_code=True) to AutoTokenizer.from_pretrained("THUDM/chatglm-6b-int4", trust_remote_code=True).
|
|
||||||
python -m pip install -r request_llms/requirements_chatglm.txt
|
|
||||||
|
|
||||||
# [Optional Step II] Support Fudan MOSS
|
|
||||||
python -m pip install -r request_llms/requirements_moss.txt
|
|
||||||
git clone --depth=1 https://github.com/OpenLMLab/MOSS.git request_llms/moss # Note: You need to be at the root directory of the project when executing this line of code
|
|
||||||
|
|
||||||
# [Optional Step III] Support RWKV Runner
|
|
||||||
Refer to the wiki: https://github.com/binary-husky/gpt_academic/wiki/%E9%80%82%E9%85%8DRWKV-Runner
|
|
||||||
|
|
||||||
# [Optional Step IV] Make sure that the AVAIL_LLM_MODELS in the config.py configuration file contains the expected models. The currently supported models are as follows (jittorllms series currently only support the docker solution):
|
|
||||||
AVAIL_LLM_MODELS = ["gpt-3.5-turbo", "api2d-gpt-3.5-turbo", "gpt-4", "api2d-gpt-4", "chatglm", "moss"] # + ["jittorllms_rwkv", "jittorllms_pangualpha", "jittorllms_llama"]
|
|
||||||
```
|
|
||||||
|
|
||||||
</p>
|
|
||||||
</details>
|
|
||||||
|
|
||||||
|
|
||||||
4. Run
|
|
||||||
```sh
|
|
||||||
python main.py
|
|
||||||
```
|
|
||||||
|
|
||||||
### Method II: Use Docker
|
|
||||||
|
|
||||||
0. Deploy all capabilities of the project (this is a large image that includes cuda and latex. But if you have a slow internet speed or a small hard drive, it is not recommended to use this)
|
|
||||||
|
|
||||||
``` sh
|
|
||||||
# Modify the docker-compose.yml file, keep scheme 0 and delete the other schemes. Then run:
|
|
||||||
docker-compose up
|
|
||||||
```
|
|
||||||
|
|
||||||
1. ChatGPT + Wenxin Yiyu + Spark and other online models (recommended for most people)
|
|
||||||
|
|
||||||
``` sh
|
|
||||||
# Modify the docker-compose.yml file, keep scheme 1 and delete the other schemes. Then run:
|
|
||||||
docker-compose up
|
|
||||||
```
|
|
||||||
|
|
||||||
NOTE: If you need Latex plugin functionality, please refer to the Wiki. Additionally, you can also use scheme 4 or scheme 0 directly to obtain Latex functionality.
|
|
||||||
|
|
||||||
2. ChatGPT + ChatGLM2 + MOSS + LLAMA2 + Tongyi Qianwen (requires familiarity with [Nvidia Docker](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html#installing-on-ubuntu-and-debian) runtime)
|
|
||||||
|
|
||||||
``` sh
|
|
||||||
# Modify the docker-compose.yml file, keep scheme 2 and delete the other schemes. Then run:
|
|
||||||
docker-compose up
|
|
||||||
```
|
|
||||||
|
|
||||||
|
|
||||||
### Method III: Other deployment methods
|
|
||||||
1. **One-click run script for Windows**.
|
|
||||||
Windows users who are completely unfamiliar with the Python environment can download the one-click run script without local models from the [Release](https://github.com/binary-husky/gpt_academic/releases) section.
|
|
||||||
The script was contributed by [oobabooga](https://github.com/oobabooga/one-click-installers).
|
|
||||||
|
|
||||||
2. Use third-party APIs, Azure, Wenxin Yiyu, Xinghuo, etc., see the [Wiki page](https://github.com/binary-husky/gpt_academic/wiki/项目配置说明).
|
|
||||||
|
|
||||||
3. Pitfall guide for deploying on cloud servers.
|
|
||||||
Please visit the [cloud server remote deployment wiki](https://github.com/binary-husky/gpt_academic/wiki/%E4%BA%91%E6%9C%8D%E5%8A%A1%E5%99%A8%E8%BF%9C%E7%A8%8B%E9%83%A8%E7%BD%B2%E6%8C%87%E5%8D%97).
|
|
||||||
|
|
||||||
4. Some new deployment platforms or methods
|
|
||||||
- Use Sealos [one-click deployment](https://github.com/binary-husky/gpt_academic/issues/993).
|
|
||||||
- Use WSL2 (Windows Subsystem for Linux). Please visit the [deployment wiki-2](https://github.com/binary-husky/gpt_academic/wiki/%E4%BD%BF%E7%94%A8WSL2%EF%BC%88Windows-Subsystem-for-Linux-%E5%AD%90%E7%B3%BB%E7%BB%9F%EF%BC%89%E9%83%A8%E7%BD%B2)
|
|
||||||
- How to run under a subpath (such as `http://localhost/subpath`). Please see [FastAPI running instructions](docs/WithFastapi.md)
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
# Utilisation avancée
|
|
||||||
### I: Personnalisation des nouveaux boutons d'accès rapide (raccourcis académiques)
|
|
||||||
Ouvrez `core_functional.py` avec n'importe quel éditeur de texte, ajoutez les entrées suivantes, puis redémarrez le programme. (Si le bouton existe déjà, le préfixe et le suffixe peuvent être modifiés à chaud sans redémarrer le programme).
|
|
||||||
Par exemple:
|
|
||||||
```
|
|
||||||
"Traduction avancée de l'anglais vers le français": {
|
|
||||||
# Préfixe, ajouté avant votre saisie. Par exemple, utilisez-le pour décrire votre demande, telle que la traduction, l'explication du code, l'amélioration, etc.
|
|
||||||
"Prefix": "Veuillez traduire le contenu suivant en français, puis expliquer chaque terme propre à la langue anglaise utilisé dans le texte à l'aide d'un tableau markdown : \n\n",
|
|
||||||
|
|
||||||
# Suffixe, ajouté après votre saisie. Par exemple, en utilisant le préfixe, vous pouvez entourer votre contenu par des guillemets.
|
|
||||||
"Suffix": "",
|
|
||||||
},
|
|
||||||
```
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/226899272-477c2134-ed71-4326-810c-29891fe4a508.png" width="500" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
### II: Personnalisation des plugins de fonction
|
|
||||||
Écrivez de puissants plugins de fonction pour accomplir toutes les tâches que vous souhaitez ou ne pouvez pas imaginer.
|
|
||||||
Le développement et le débogage de ces plugins dans ce projet sont très faciles. Tant que vous avez des connaissances de base en python, vous pouvez implémenter vos propres fonctionnalités grâce à notre modèle fourni.
|
|
||||||
Veuillez consulter le [Guide des plugins de fonction](https://github.com/binary-husky/gpt_academic/wiki/%E5%87%BD%E6%95%B0%E6%8F%92%E4%BB%B6%E6%8C%87%E5%8D%97) pour plus de détails.
|
|
||||||
|
|
||||||
|
|
||||||
# Mises à jour
|
|
||||||
### I: Dynamique
|
|
||||||
|
|
||||||
1. Fonction de sauvegarde de conversation. Appelez `Enregistrer la conversation en cours` dans la zone des plugins fonctionnels pour enregistrer la conversation en cours sous la forme d'un fichier HTML lisible et récupérable. En outre, appelez `Charger les archives de conversation` dans la zone des plugins fonctionnels (menu déroulant) pour restaurer les conversations précédentes.
|
|
||||||
Astuce: Si aucun fichier n'est spécifié, cliquez directement sur `Charger les archives de conversation` pour afficher le cache des archives HTML.
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/235222390-24a9acc0-680f-49f5-bc81-2f3161f1e049.png" width="500" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
2. ⭐ Fonction de traduction des articles Latex/Arxiv ⭐
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://github.com/binary-husky/gpt_academic/assets/96192199/002a1a75-ace0-4e6a-94e2-ec1406a746f1" height="250" > ===>
|
|
||||||
<img src="https://github.com/binary-husky/gpt_academic/assets/96192199/9fdcc391-f823-464f-9322-f8719677043b" height="250" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
3. Terminal du néant (comprendre l'intention de l'utilisateur à partir de la saisie en langage naturel et appeler automatiquement d'autres plugins)
|
|
||||||
|
|
||||||
- Étape 1: Saisissez "Veuillez appeler le plugin de traduction pour le document PDF, l'URL est https://openreview.net/pdf?id=rJl0r3R9KX".
|
|
||||||
- Étape 2 : Cliquez sur "Terminal du néant".
|
|
||||||
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://github.com/binary-husky/gpt_academic/assets/96192199/66f1b044-e9ff-4eed-9126-5d4f3668f1ed" width="500" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
4. Conception de fonctionnalités modulaires, une interface simple peut prendre en charge des fonctionnalités puissantes
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/229288270-093643c1-0018-487a-81e6-1d7809b6e90f.png" height="400" >
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/227504931-19955f78-45cd-4d1c-adac-e71e50957915.png" height="400" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
5. Traduction et interprétation d'autres projets open-source
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/226935232-6b6a73ce-8900-4aee-93f9-733c7e6fef53.png" height="250" >
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/226969067-968a27c1-1b9c-486b-8b81-ab2de8d3f88a.png" height="250" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
6. Fonctionnalités supplémentaires intégrant [live2d](https://github.com/fghrsh/live2d_demo) (désactivé par défaut, nécessite des modifications dans `config.py`)
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/236432361-67739153-73e8-43fe-8111-b61296edabd9.png" width="500" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
7. Génération d'images par OpenAI
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://github.com/binary-husky/gpt_academic/assets/96192199/bc7ab234-ad90-48a0-8d62-f703d9e74665" width="500" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
8. Analyse et résumé audio par OpenAI
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://github.com/binary-husky/gpt_academic/assets/96192199/709ccf95-3aee-498a-934a-e1c22d3d5d5b" width="500" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
9. Vérification et correction orthographique complète du document en Latex
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://github.com/binary-husky/gpt_academic/assets/96192199/651ccd98-02c9-4464-91e1-77a6b7d1b033" height="200" > ===>
|
|
||||||
<img src="https://github.com/binary-husky/gpt_academic/assets/96192199/476f66d9-7716-4537-b5c1-735372c25adb" height="200">
|
|
||||||
</div>
|
|
||||||
|
|
||||||
10. Changement de langue et de thème
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://github.com/binary-husky/gpt_academic/assets/96192199/b6799499-b6fb-4f0c-9c8e-1b441872f4e8" width="500" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
### II: Versions:
|
|
||||||
- version 3.70(tâche à accomplir) : Optimisation de la fonction AutoGen et création d'une série de plugins dérivés
|
|
||||||
- version 3.60 : Introduction d'AutoGen comme base des nouveaux plugins
|
|
||||||
- version 3.57 : Prise en charge de GLM3, Starlight v3, Zen v4 et correction de l'incompatibilité des modèles locaux
|
|
||||||
- version 3.56 : Possibilité d'ajouter dynamiquement des boutons de fonction de base et nouvelle page de synthèse des PDF
|
|
||||||
- version 3.55: Refonte de l'interface utilisateur avec fenêtres flottantes et barre de menu
|
|
||||||
- version 3.54 : Nouvel interpréteur de code dynamique (Code Interpreter) (à améliorer)
|
|
||||||
- version 3.53 : Possibilité de choisir dynamiquement différents thèmes d'interface, amélioration de la stabilité et résolution des problèmes de conflit entre utilisateurs multiples
|
|
||||||
- version 3.50 : Utiliser le langage naturel pour appeler toutes les fonctions du projet (Terminal du néant), prise en charge de la classification des plugins, amélioration de l'interface utilisateur, conception de nouveaux thèmes
|
|
||||||
- version 3.49 : Prise en charge de Baidu Qianfan et Xiaomi-Wenyiyan
|
|
||||||
- version 3.48 : Prise en charge d'Ali-DA, Shanghai AI-Lab-Shusheng et Xunfei Xinghuo
|
|
||||||
- version 3.46 : Prise en charge de la conversation audio temps réel sans intervention
|
|
||||||
- version 3.45 : Prise en charge de la personnalisation du modèle ChatGLM2
|
|
||||||
- version 3.44 : Prise en charge officielle d'Azure, amélioration de l'utilisabilité de l'interface
|
|
||||||
- version 3.4 : +traduction complète des articles Arxiv, +correction des articles Latex
|
|
||||||
- version 3.3 : +fonction d'intégration d'informations Internet
|
|
||||||
- version 3.2 : Les plugins de fonction prennent en charge plus de paramètres (fonction d'enregistrement de conversation, débogage de code de n'importe quel langage + demandes d'LLM arbitraires)
|
|
||||||
- version 3.1 : Prise en charge de l'interrogation simultanée de plusieurs modèles gpt ! Prise en charge de l'API2D, répartition de charge entre plusieurs clés API
|
|
||||||
- version 3.0 : Prise en charge de chatglm et d'autres petits llm
|
|
||||||
- version 2.6 : Refonte de la structure des plugins, amélioration de l'interactivité, ajout de nouveaux plugins
|
|
||||||
- version 2.5 : Auto-mise à jour, résolution des problèmes de dépassement de longueur de texte et de jeton pendant la consolidation de grands projets de codes sources
|
|
||||||
- version 2.4 : (1) Nouvelle fonctionnalité de traduction complète des documents PDF ; (2) Nouvelle fonctionnalité de changement de position de la zone de saisie ; (3) Nouvelle option de disposition verticale ; (4) Optimisation des plugins de fonction multithreads.
|
|
||||||
- version 2.3 : Amélioration de l'interactivité multi-threads
|
|
||||||
- version 2.2 : Prise en charge du rechargement à chaud des plugins de fonction
|
|
||||||
- version 2.1 : Mise en page pliable
|
|
||||||
- version 2.0 : Introduction de plugins de fonction modulaires
|
|
||||||
- version 1.0: Fonctionnalités de base
|
|
||||||
|
|
||||||
Groupe QQ des développeurs de GPT Academic: `610599535`
|
|
||||||
|
|
||||||
- Problèmes connus
|
|
||||||
- Certains plugins de traduction de navigateurs peuvent nuire au fonctionnement de l'interface utilisateur de ce logiciel.
|
|
||||||
- Gradio officiel a actuellement de nombreux bugs de compatibilité. Veuillez utiliser `requirement.txt` pour installer Gradio.
|
|
||||||
|
|
||||||
### III: Thèmes
|
|
||||||
Vous pouvez modifier le thème en modifiant l'option `THEME` (config.py).
|
|
||||||
|
|
||||||
1. `Chuanhu-Small-and-Beautiful` [Lien](https://github.com/GaiZhenbiao/ChuanhuChatGPT/)
|
|
||||||
|
|
||||||
|
|
||||||
### IV: Branches de développement de ce projet
|
|
||||||
|
|
||||||
1. Branche `master` : Branche principale, version stable
|
|
||||||
2. Branche `frontier` : Branche de développement, version de test
|
|
||||||
|
|
||||||
|
|
||||||
### V: Références et apprentissage
|
|
||||||
|
|
||||||
```
|
|
||||||
De nombreux designs de codes de projets exceptionnels ont été référencés dans le développement de ce projet, sans ordre spécifique :
|
|
||||||
|
|
||||||
# ChatGLM2-6B de l'Université Tsinghua:
|
|
||||||
https://github.com/THUDM/ChatGLM2-6B
|
|
||||||
|
|
||||||
# JittorLLMs de l'Université Tsinghua:
|
|
||||||
https://github.com/Jittor/JittorLLMs
|
|
||||||
|
|
||||||
# ChatPaper :
|
|
||||||
https://github.com/kaixindelele/ChatPaper
|
|
||||||
|
|
||||||
# Edge-GPT :
|
|
||||||
https://github.com/acheong08/EdgeGPT
|
|
||||||
|
|
||||||
# ChuanhuChatGPT :
|
|
||||||
https://github.com/GaiZhenbiao/ChuanhuChatGPT
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
# Oobabooga installeur en un clic :
|
|
||||||
https://github.com/oobabooga/one-click-installers
|
|
||||||
|
|
||||||
# Plus：
|
|
||||||
https://github.com/gradio-app/gradio
|
|
||||||
https://github.com/fghrsh/live2d_demo
|
|
||||||
@@ -1,363 +0,0 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
> **Hinweis**
|
|
||||||
>
|
|
||||||
> Dieses README wurde mithilfe der GPT-Übersetzung (durch das Plugin dieses Projekts) erstellt und ist nicht zu 100 % zuverlässig. Bitte überprüfen Sie die Übersetzungsergebnisse sorgfältig.
|
|
||||||
>
|
|
||||||
> 7. November 2023: Beim Installieren der Abhängigkeiten bitte nur die in der `requirements.txt` **angegebenen Versionen** auswählen. Installationsbefehl: `pip install -r requirements.txt`.
|
|
||||||
|
|
||||||
|
|
||||||
# <div align=center><img src="logo.png" width="40"> GPT Academic (GPT Akademisch)</div>
|
|
||||||
|
|
||||||
**Wenn Ihnen dieses Projekt gefällt, geben Sie ihm bitte einen Star. Wenn Sie praktische Tastenkombinationen oder Plugins entwickelt haben, sind Pull-Anfragen willkommen!**
|
|
||||||
|
|
||||||
Wenn Ihnen dieses Projekt gefällt, geben Sie ihm bitte einen Star.
|
|
||||||
Um dieses Projekt mit GPT in eine beliebige Sprache zu übersetzen, lesen Sie [`multi_language.py`](multi_language.py) (experimentell).
|
|
||||||
|
|
||||||
> **Hinweis**
|
|
||||||
>
|
|
||||||
> 1. Beachten Sie bitte, dass nur die mit **hervorgehobenen** Plugins (Schaltflächen) Dateien lesen können. Einige Plugins befinden sich im **Drop-down-Menü** des Plugin-Bereichs. Außerdem freuen wir uns über jede neue Plugin-PR mit **höchster Priorität**.
|
|
||||||
>
|
|
||||||
> 2. Die Funktionen jeder Datei in diesem Projekt sind im [Selbstanalysebericht `self_analysis.md`](https://github.com/binary-husky/gpt_academic/wiki/GPT-Academic-Selbstanalysebericht) ausführlich erläutert. Sie können jederzeit auf die relevanten Funktions-Plugins klicken und GPT aufrufen, um den Selbstanalysebericht des Projekts neu zu generieren. Häufig gestellte Fragen finden Sie im [`Wiki`](https://github.com/binary-husky/gpt_academic/wiki). [Standardinstallationsmethode](#installation) | [Ein-Klick-Installationsskript](https://github.com/binary-husky/gpt_academic/releases) | [Konfigurationsanleitung](https://github.com/binary-husky/gpt_academic/wiki/Projekt-Konfigurationsanleitung).
|
|
||||||
>
|
|
||||||
> 3. Dieses Projekt ist kompatibel mit und unterstützt auch die Verwendung von inländischen Sprachmodellen wie ChatGLM. Die gleichzeitige Verwendung mehrerer API-Schlüssel ist möglich, indem Sie sie in der Konfigurationsdatei wie folgt angeben: `API_KEY="openai-key1,openai-key2,azure-key3,api2d-key4"`. Wenn Sie den `API_KEY` vorübergehend ändern möchten, geben Sie vorübergehend den temporären `API_KEY` im Eingabebereich ein und drücken Sie die Eingabetaste, um die Änderung wirksam werden zu lassen.
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
<div align="center">
|
|
||||||
|
|
||||||
Funktionen (⭐= Kürzlich hinzugefügte Funktion) | Beschreibung
|
|
||||||
--- | ---
|
|
||||||
⭐[Neues Modell integrieren](https://github.com/binary-husky/gpt_academic/wiki/%E5%A6%82%E4%BD%95%E5%88%87%E6%8D%A2%E6%A8%A1%E5%9E%8B)! | Baidu [Qianfan](https://cloud.baidu.com/doc/WENXINWORKSHOP/s/Nlks5zkzu) und Wenxin Yanyi, [Tongyi Qianwen](https://modelscope.cn/models/qwen/Qwen-7B-Chat/summary), Shanghai AI-Lab [Shusheng](https://github.com/InternLM/InternLM), Xunfei [Xinghuo](https://xinghuo.xfyun.cn/), [LLaMa2](https://huggingface.co/meta-llama/Llama-2-7b-chat-hf), Cognitive Graph API, DALLE3
|
|
||||||
Verfeinern, Übersetzen, Codierung erläutern | Ein-Klick-Verfeinerung, Übersetzung, Suche nach grammatikalischen Fehlern in wissenschaftlichen Arbeiten, Erklärung von Code
|
|
||||||
[Eigene Tastenkombinationen](https://www.bilibili.com/video/BV14s4y1E7jN) definieren | Eigene Tastenkombinationen definieren
|
|
||||||
Modulare Gestaltung | Ermöglicht die Verwendung benutzerdefinierter leistungsstarker [Plugins](https://github.com/binary-husky/gpt_academic/tree/master/crazy_functions), Plugins unterstützen [Hot-Reload](https://github.com/binary-husky/gpt_academic/wiki/%E5%87%BD%E6%95%B0%E6%8F%92%E4%BB%B6%E6%8C%87%E5%8D%97)
|
|
||||||
[Programmanalyse](https://www.bilibili.com/video/BV1cj411A7VW) | [Plugin] Ermöglicht die Erstellung einer Projekthierarchie für Python/C/C++/Java/Lua/... mit nur einem Klick oder [Selbstanalyse](https://www.bilibili.com/video/BV1cj411A7VW)
|
|
||||||
Lesen von Forschungsarbeiten, Übersetzen von Forschungsarbeiten | [Plugin] Ermöglicht eine Umwandlung des gesamten Latex-/PDF-Forschungspapiers mit nur einem Klick und generiert eine Zusammenfassung
|
|
||||||
Latex-Übersetzung des vollständigen Textes, Ausbesserung | [Plugin] Ermöglicht eine Übersetzung oder Verbesserung der Latex-Forschungsarbeit mit nur einem Klick
|
|
||||||
Erzeugen von Batch-Anmerkungen | [Plugin] Erzeugt Funktionserläuterungen in Stapeln
|
|
||||||
Markdown- [En-De-Übersetzung](https://www.bilibili.com/video/BV1yo4y157jV/) | [Plugin] Haben Sie die [README](https://github.com/binary-husky/gpt_academic/blob/master/docs/README_EN.md) in den oben genannten 5 Sprachen gesehen?
|
|
||||||
Erzeugen eines Chat-Analyseberichts | [Plugin] Generiert einen zusammenfassenden Bericht nach der Ausführung
|
|
||||||
PDF-Textübersetzungsmerkmal | [Plugin] Extrahiert Titel und Zusammenfassung des PDF-Dokuments und übersetzt den vollständigen Text (mehrfädig)
|
|
||||||
Arxiv-Assistent | [Plugin] Geben Sie die URL eines Arxiv-Artikels ein, um eine Zusammenfassung zu übersetzen und die PDF-Datei herunterzuladen
|
|
||||||
Automatische Überprüfung von Latex-Artikeln | [Plugin] Überprüft die Grammatik und Rechtschreibung von Latex-Artikeln nach dem Vorbild von Grammarly und generiert eine PDF-Vergleichsdatei
|
|
||||||
Google Scholar Integration Assistant | [Plugin] Geben Sie eine beliebige URL der Google Scholar-Suchseite ein und lassen Sie GPT Ihre [Verwandten Arbeiten](https://www.bilibili.com/video/BV1GP411U7Az/) schreiben
|
|
||||||
Internetinformationsaggregation + GPT | [Plugin] Ermöglicht es GPT, Fragen durch das Durchsuchen des Internets zu beantworten und Informationen immer auf dem neuesten Stand zu halten
|
|
||||||
⭐Feine Übersetzung von Arxiv-Artikeln ([Docker](https://github.com/binary-husky/gpt_academic/pkgs/container/gpt_academic_with_latex)) | [Plugin] Übersetzt Arxiv-Artikel [mit hoher Qualität](https://www.bilibili.com/video/BV1dz4y1v77A/) mit einem Klick - das beste Übersetzungstool für wissenschaftliche Artikel
|
|
||||||
⭐[Echtzeit-Spracheingabe](https://github.com/binary-husky/gpt_academic/blob/master/docs/use_audio.md) | [Plugin] [Asynchrones Lauschen auf Audio-Eingabe](https://www.bilibili.com/video/BV1AV4y187Uy/), automatisches Zerschneiden des Textes, automatische Suche nach dem richtigen Zeitpunkt zur Beantwortung
|
|
||||||
Darstellen von Formeln/Bildern/Tabellen | Zeigt Formeln sowohl in [TEX-](https://user-images.githubusercontent.com/96192199/230598842-1d7fcddd-815d-40ee-af60-baf488a199df.png)- als auch in gerenderten Formen an, unterstützt Formeln und Code-Hervorhebung
|
|
||||||
⭐AutoGen Multi-Agent Plugin | [Plugin] Erforscht die Möglichkeiten des emergenten Verhaltens von Multi-Agent-Systemen mit Microsoft AutoGen!
|
|
||||||
Start im Dark-Theme | Um das Dark-Theme zu aktivieren, fügen Sie ```/?__theme=dark``` am Ende der URL im Browser hinzu
|
|
||||||
[Mehrsprachige LLM-Modelle](https://www.bilibili.com/video/BV1wT411p7yf) unterstützt | Es ist sicherlich beeindruckend, von GPT3.5, GPT4, [ChatGLM2 der Tsinghua University](https://github.com/THUDM/ChatGLM2-6B), [MOSS der Fudan University](https://github.com/OpenLMLab/MOSS) bedient zu werden, oder?
|
|
||||||
⭐ChatGLM2 Feinabstimmungsmodell | Unterstützt das Laden von ChatGLM2-Feinabstimmungsmodellen und bietet Unterstützung für ChatGLM2-Feinabstimmungsassistenten
|
|
||||||
Integration weiterer LLM-Modelle, Unterstützung von [Huggingface-Deployment](https://huggingface.co/spaces/qingxu98/gpt-academic) | Hinzufügen der Newbing-Schnittstelle (neues Bing), Einführung der [Jittorllms der Tsinghua University](https://github.com/Jittor/JittorLLMs) zur Unterstützung von LLaMA und PanGu Alpha
|
|
||||||
⭐[void-terminal](https://github.com/binary-husky/void-terminal) Pip-Paket | Verwenden Sie das Projekt in Python direkt, indem Sie das gesamte Funktionsplugin verwenden (in Entwicklung)
|
|
||||||
⭐Void-Terminal-Plugin | [Plugin] Verwenden Sie natürliche Sprache, um andere Funktionen dieses Projekts direkt zu steuern
|
|
||||||
Weitere Funktionen anzeigen (z. B. Bildgenerierung) …… | Siehe das Ende dieses Dokuments ……
|
|
||||||
</div>
|
|
||||||
|
|
||||||
|
|
||||||
- Neues Interface (Ändern Sie die LAYOUT-Option in der `config.py`, um zwischen "Links-Rechts-Layout" und "Oben-Unten-Layout" zu wechseln)
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://github.com/binary-husky/gpt_academic/assets/96192199/d81137c3-affd-4cd1-bb5e-b15610389762" width="700" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
|
|
||||||
- Alle Schaltflächen werden dynamisch aus der `functional.py` generiert und ermöglichen das beliebige Hinzufügen benutzerdefinierter Funktionen zur Befreiung der Zwischenablage.
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/231975334-b4788e91-4887-412f-8b43-2b9c5f41d248.gif" width="700" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
- Überarbeiten/Korrigieren
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/231980294-f374bdcb-3309-4560-b424-38ef39f04ebd.gif" width="700" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
- If the output contains formulas, they will be displayed in both tex format and rendering format for easy copying and reading.
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/230598842-1d7fcddd-815d-40ee-af60-baf488a199df.png" width="700" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
- Don't want to look at the project code? Show off the whole project directly in chatgpt's mouth.
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/226935232-6b6a73ce-8900-4aee-93f9-733c7e6fef53.png" width="700" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
- Multiple large language models mixed calling (ChatGLM + OpenAI-GPT3.5 + [API2D](https://api2d.com/)-GPT4)
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/232537274-deca0563-7aa6-4b5d-94a2-b7c453c47794.png" width="700" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
# Installation
|
|
||||||
### Installation Method I: Run directly (Windows, Linux or MacOS)
|
|
||||||
|
|
||||||
1. Download the project
|
|
||||||
```sh
|
|
||||||
git clone --depth=1 https://github.com/binary-husky/gpt_academic.git
|
|
||||||
cd gpt_academic
|
|
||||||
```
|
|
||||||
|
|
||||||
2. Configure API_KEY
|
|
||||||
|
|
||||||
In `config.py`, configure API KEY and other settings, [click to view special network environment configuration methods](https://github.com/binary-husky/gpt_academic/issues/1). [Wiki page](https://github.com/binary-husky/gpt_academic/wiki/Project-Configuration-Instructions).
|
|
||||||
|
|
||||||
「 The program will first check if there is a confidential configuration file named `config_private.py` and use its configuration to override the configuration with the same name in `config.py`. If you understand this reading logic, we strongly recommend that you create a new configuration file named `config_private.py` next to `config.py` and move (copy) the configuration in `config.py` to `config_private.py` (only copy the configuration items that you have modified). 」
|
|
||||||
|
|
||||||
「 You can configure the project through `environment variables`. The format of environment variables can refer to the `docker-compose.yml` file or our [Wiki page](https://github.com/binary-husky/gpt_academic/wiki/Project-Configuration-Instructions). The priority of configuration reading is: `environment variables` > `config_private.py` > `config.py`. 」
|
|
||||||
|
|
||||||
|
|
||||||
3. Install dependencies
|
|
||||||
```sh
|
|
||||||
# (Option I: if you are familiar with python, python>=3.9) Note: Use the official pip source or Ali pip source, temporary method to change the source: python -m pip install -r requirements.txt -i https://mirrors.aliyun.com/pypi/simple/
|
|
||||||
python -m pip install -r requirements.txt
|
|
||||||
|
|
||||||
# (Option II: Using Anaconda) The steps are similar (https://www.bilibili.com/video/BV1rc411W7Dr):
|
|
||||||
conda create -n gptac_venv python=3.11 # Create an anaconda environment
|
|
||||||
conda activate gptac_venv # Activate the anaconda environment
|
|
||||||
python -m pip install -r requirements.txt # This step is the same as installing with pip
|
|
||||||
```
|
|
||||||
|
|
||||||
|
|
||||||
<details><summary>If you need support for Tsinghua ChatGLM2/Fudan MOSS/RWKV as backend, please click to expand.</summary>
|
|
||||||
<p>
|
|
||||||
|
|
||||||
[Optional] If you need to support Tsinghua ChatGLM2/Fudan MOSS as the backend, you need to install additional dependencies (Prerequisites: Familiar with Python + Have used PyTorch + Strong computer configuration):
|
|
||||||
```sh
|
|
||||||
# [Optional Step I] Support Tsinghua ChatGLM2. Tsinghua ChatGLM note: If you encounter the error "Call ChatGLM fail cannot load ChatGLM parameters normally", refer to the following: 1: The default installation above is torch+cpu version. To use cuda, you need to uninstall torch and reinstall torch+cuda; 2: If you cannot load the model due to insufficient computer configuration, you can modify the model accuracy in request_llm/bridge_chatglm.py. Change AutoTokenizer.from_pretrained("THUDM/chatglm-6b", trust_remote_code=True) to AutoTokenizer.from_pretrained("THUDM/chatglm-6b-int4", trust_remote_code=True)
|
|
||||||
python -m pip install -r request_llms/requirements_chatglm.txt
|
|
||||||
|
|
||||||
# [Optional Step II] Support Fudan MOSS
|
|
||||||
python -m pip install -r request_llms/requirements_moss.txt
|
|
||||||
git clone --depth=1 https://github.com/OpenLMLab/MOSS.git request_llms/moss # When executing this line of code, you must be in the root path of the project
|
|
||||||
|
|
||||||
# [Optional Step III] Support RWKV Runner
|
|
||||||
Refer to the wiki: https://github.com/binary-husky/gpt_academic/wiki/Support-RWKV-Runner
|
|
||||||
|
|
||||||
# [Optional Step IV] Make sure the AVAIL_LLM_MODELS in config.py includes the expected models. The currently supported models are as follows (the jittorllms series only supports the docker solution at present):
|
|
||||||
AVAIL_LLM_MODELS = ["gpt-3.5-turbo", "api2d-gpt-3.5-turbo", "gpt-4", "api2d-gpt-4", "chatglm", "moss"] # + ["jittorllms_rwkv", "jittorllms_pangualpha", "jittorllms_llama"]
|
|
||||||
```
|
|
||||||
|
|
||||||
</p>
|
|
||||||
</details>
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
4. Run
|
|
||||||
```sh
|
|
||||||
python main.py
|
|
||||||
```
|
|
||||||
|
|
||||||
### Installation Method II: Use Docker
|
|
||||||
|
|
||||||
0. Deploy all capabilities of the project (this is a large image that includes cuda and latex. But if you have a slow internet speed or a small hard drive, it is not recommended to use this)
|
|
||||||
[](https://github.com/binary-husky/gpt_academic/actions/workflows/build-with-all-capacity.yml)
|
|
||||||
|
|
||||||
``` sh
|
|
||||||
# Modify docker-compose.yml, keep solution 0 and delete other solutions. Then run:
|
|
||||||
docker-compose up
|
|
||||||
```
|
|
||||||
|
|
||||||
1. ChatGPT + Wenxin's words + spark and other online models (recommended for most people)
|
|
||||||
[](https://github.com/binary-husky/gpt_academic/actions/workflows/build-without-local-llms.yml)
|
|
||||||
[](https://github.com/binary-husky/gpt_academic/actions/workflows/build-with-latex.yml)
|
|
||||||
[](https://github.com/binary-husky/gpt_academic/actions/workflows/build-with-audio-assistant.yml)
|
|
||||||
|
|
||||||
``` sh
|
|
||||||
# Modify docker-compose.yml, keep solution 1 and delete other solutions. Then run:
|
|
||||||
docker-compose up
|
|
||||||
```
|
|
||||||
|
|
||||||
P.S. If you need the Latex plugin functionality, please refer to the Wiki. Also, you can directly use solution 4 or 0 to get the Latex functionality.
|
|
||||||
|
|
||||||
2. ChatGPT + ChatGLM2 + MOSS + LLAMA2 + Thousand Questions (Requires familiarity with [Nvidia Docker](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html#installing-on-ubuntu-and-debian) runtime)
|
|
||||||
[](https://github.com/binary-husky/gpt_academic/actions/workflows/build-with-chatglm.yml)
|
|
||||||
|
|
||||||
``` sh
|
|
||||||
# Modify docker-compose.yml, keep solution 2 and delete other solutions. Then run:
|
|
||||||
docker-compose up
|
|
||||||
```
|
|
||||||
|
|
||||||
|
|
||||||
### Installation Method III: Other Deployment Methods
|
|
||||||
1. **Windows One-Click Script**.
|
|
||||||
Windows users who are completely unfamiliar with the python environment can download the one-click script for installation without local models in the published [Release](https://github.com/binary-husky/gpt_academic/releases).
|
|
||||||
The script is contributed by [oobabooga](https://github.com/oobabooga/one-click-installers).
|
|
||||||
|
|
||||||
2. Use third-party APIs, Azure, Wenxin's words, Spark, etc., see [Wiki page](https://github.com/binary-husky/gpt_academic/wiki/Project-Configuration-Instructions)
|
|
||||||
|
|
||||||
3. Pit avoidance guide for cloud server remote deployment.
|
|
||||||
Please visit the [Cloud Server Remote Deployment Wiki](https://github.com/binary-husky/gpt_academic/wiki/Cloud-Server-Remote-Deployment-Guide)
|
|
||||||
|
|
||||||
4. Some new deployment platforms or methods
|
|
||||||
- Use Sealos [one-click deployment](https://github.com/binary-husky/gpt_academic/issues/993).
|
|
||||||
- Use WSL2 (Windows Subsystem for Linux). Please visit the [deployment wiki-2](https://github.com/binary-husky/gpt_academic/wiki/Deploy-on-Windows-Subsystem-for-Linux-WSL2)
|
|
||||||
- How to run under a subpath (such as `http://localhost/subpath`). Please visit [FastAPI Running Instructions](docs/WithFastapi.md)
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
# Fortgeschrittene Nutzung
|
|
||||||
### I: Benutzerdefinierte Tasten hinzufügen (akademische Hotkeys)
|
|
||||||
Öffnen Sie die Datei `core_functional.py` mit einem beliebigen Texteditor und fügen Sie folgenden Eintrag hinzu. Starten Sie dann das Programm neu. (Wenn die Schaltfläche bereits vorhanden ist, können sowohl das Präfix als auch das Suffix schnell geändert werden, ohne dass das Programm neu gestartet werden muss.)
|
|
||||||
|
|
||||||
Beispiel:
|
|
||||||
```
|
|
||||||
"Übersetzung von Englisch nach Chinesisch": {
|
|
||||||
# Präfix, wird vor Ihrer Eingabe hinzugefügt. Zum Beispiel, um Ihre Anforderungen zu beschreiben, z.B. Übersetzen, Code erklären, verbessern usw.
|
|
||||||
"Präfix": "Bitte übersetzen Sie den folgenden Abschnitt ins Chinesische und erklären Sie dann jedes Fachwort in einer Markdown-Tabelle:\n\n",
|
|
||||||
|
|
||||||
# Suffix, wird nach Ihrer Eingabe hinzugefügt. Zum Beispiel, um Ihre Eingabe in Anführungszeichen zu setzen.
|
|
||||||
"Suffix": "",
|
|
||||||
},
|
|
||||||
```
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/226899272-477c2134-ed71-4326-810c-29891fe4a508.png" width="500" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
### II: Benutzerdefinierte Funktionsplugins
|
|
||||||
Schreiben Sie leistungsstarke Funktionsplugins, um beliebige Aufgaben zu erledigen, die Sie wünschen oder nicht erwartet haben.
|
|
||||||
Das Erstellen und Debuggen von Plugins in diesem Projekt ist einfach und erfordert nur Grundkenntnisse in Python. Sie können unser bereitgestelltes Template verwenden, um Ihre eigene Plugin-Funktion zu implementieren.
|
|
||||||
Weitere Informationen finden Sie in der [Plugin-Anleitung](https://github.com/binary-husky/gpt_academic/wiki/%E5%87%BD%E6%95%B0%E6%8F%92%E4%BB%B6%E6%8C%87%E5%8D%97).
|
|
||||||
|
|
||||||
|
|
||||||
# Aktualisierungen
|
|
||||||
### I: Neuigkeiten
|
|
||||||
|
|
||||||
1. Dialogspeicherungsfunktion. Rufen Sie im Funktionspluginbereich "Aktuellen Dialog speichern" auf, um den aktuellen Dialog als lesbare und wiederherstellbare HTML-Datei zu speichern.
|
|
||||||
Darüber hinaus können Sie im Funktionspluginbereich (Dropdown-Menü) "Dialoghistorie laden" aufrufen, um frühere Sitzungen wiederherzustellen.
|
|
||||||
Tipp: Wenn kein Dateiname angegeben ist, können Sie direkt auf "Dialoghistorie laden" klicken, um den Verlauf des HTML-Archivs anzuzeigen.
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/235222390-24a9acc0-680f-49f5-bc81-2f3161f1e049.png" width="500" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
2. ⭐ Latex/Arxiv-Papierübersetzungsfunktion ⭐
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://github.com/binary-husky/gpt_academic/assets/96192199/002a1a75-ace0-4e6a-94e2-ec1406a746f1" height="250" > ===>
|
|
||||||
<img src="https://github.com/binary-husky/gpt_academic/assets/96192199/9fdcc391-f823-464f-9322-f8719677043b" height="250" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
3. Leere Terminaloberfläche (Verständnis der Benutzerabsicht und automatischer Aufruf anderer Plugins aus natürlicher Spracheingabe)
|
|
||||||
|
|
||||||
- Schritt 1: Geben Sie "Bitte Plugin aufrufen, um das PDF-Papier zu übersetzen, dessen Adresse https://openreview.net/pdf?id=rJl0r3R9KX ist" ein.
|
|
||||||
- Schritt 2: Klicken Sie auf "Leere Terminaloberfläche".
|
|
||||||
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://github.com/binary-husky/gpt_academic/assets/96192199/66f1b044-e9ff-4eed-9126-5d4f3668f1ed" width="500" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
4. Modulare Funktionsgestaltung mit einfacher Schnittstelle für leistungsstarke Funktionen
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/229288270-093643c1-0018-487a-81e6-1d7809b6e90f.png" height="400" >
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/227504931-19955f78-45cd-4d1c-adac-e71e50957915.png" height="400" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
5. Übersetzung und Lösung anderer Open-Source-Projekte
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/226935232-6b6a73ce-8900-4aee-93f9-733c7e6fef53.png" height="250" >
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/226969067-968a27c1-1b9c-486b-8b81-ab2de8d3f88a.png" height="250" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
6. Funktionen zur Dekoration von [live2d](https://github.com/fghrsh/live2d_demo) (standardmäßig deaktiviert, config.py muss geändert werden)
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/236432361-677397d7-abe6-4358-9bef-3c14a7041b59.png" width="500" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
7. OpenAI-Bildgenerierung
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://github.com/binary-husky/gpt_academic/assets/96192199/bc7ab234-ad90-48a0-8d62-f703d9e74665" width="500" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
8. OpenAI-Audioanalyse und Zusammenfassung
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://github.com/binary-husky/gpt_academic/assets/96192199/709ccf95-3aee-498a-934a-e1c22d3d5d5b" width="500" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
9. Latex-Volltextkorrektur
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://github.com/binary-husky/gpt_academic/assets/96192199/651ccd98-02c9-4464-91e1-77a6b7d1b033" height="200" > ===>
|
|
||||||
<img src="https://github.com/binary-husky/gpt_academic/assets/96192199/476f66d9-7716-4537-b5c1-735372c25adb" height="200">
|
|
||||||
</div>
|
|
||||||
|
|
||||||
10. Sprach- und Themenwechsel
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://github.com/binary-husky/gpt_academic/assets/96192199/b6799499-b6fb-4f0c-9c8e-1b441872f4e8" width="500" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
### II: Versionen:
|
|
||||||
- Version 3.70 (ausstehend): Optimierung des AutoGen-Plugin-Themas und Entwicklung einer Reihe von abgeleiteten Plugins
|
|
||||||
- Version 3.60: Einführung von AutoGen als Grundlage für neue Plugin-Generation
|
|
||||||
- Version 3.57: Unterstützung von GLM3, SparkV3, WenxinYiyanV4, Behebung von Problemen bei gleichzeitiger Verwendung von lokalen Modellen
|
|
||||||
- Version 3.56: Dynamische Hinzufügung von Basisfunktionsbuttons, neue Übersichtsseite für PDFs
|
|
||||||
- Version 3.55: Überarbeitung der Benutzeroberfläche, Hinzufügung von Schwebefenstern und Menüleiste
|
|
||||||
- Version 3.54: Neuer dynamischer Code interpretier (Code Interpreter) (unfertig)
|
|
||||||
- Version 3.53: Unterstützung für dynamische Auswahl verschiedener Oberflächenthemen, Verbesserung der Stabilität und Behebung von Mehrbenutzerkonflikten
|
|
||||||
- Version 3.50: Verwenden Sie natürliche Sprache, um alle Funktionen dieses Projekts aufzurufen (leeres Terminal), Unterstützung für Plugin-Kategorien, verbesserte Benutzeroberfläche, neue Themen
|
|
||||||
- Version 3.49: Unterstützung für Baidu Qianfan Platform und WenxinYiyan
|
|
||||||
- Version 3.48: Unterstützung für Alibaba Damo Academy Tongyi Qianwen, Shanghai AI-Lab Shusheng, Xunfei Spark
|
|
||||||
- Version 3.46: Vollständig automatisierter Echtzeit-Sprachdialog
|
|
||||||
- Version 3.45: Anpassbare ChatGLM2-Feinjustierung
|
|
||||||
- Version 3.44: Offizielle Unterstützung für Azure, Verbesserung der Benutzerfreundlichkeit der Benutzeroberfläche
|
|
||||||
- Version 3.4: Hinzufügen von Arxiv-Papierübersetzung, LaTeX-Papierkorrektur
|
|
||||||
- Version 3.3: Hinzufügen von Internet-Informationen
|
|
||||||
- Version 3.2: Funktionsplugins unterstützen weitere Parameter (Dialog speichern, beliebigen Code analysieren und nach beliebigen LLM-Kombinationen fragen)
|
|
||||||
- Version 3.1: Unterstützung für die gleichzeitige Abfrage mehrerer GPT-Modelle! Unterstützung für API-Schlüssel-Lastenausgleich
|
|
||||||
- Version 3.0: Unterstützung von ChatGLM und anderen kleinen LLMs
|
|
||||||
- Version 2.6: Neugestaltung der Plugin-Struktur, Verbesserung der Interaktivität, Hinzufügen weiterer Plugins
|
|
||||||
- Version 2.5: Auto-Update zur Lösung von Problemen mit zu langem Text oder Tokenüberschuss beim Zusammenfassen von Code
|
|
||||||
- Version 2.4: (1) Hinzufügen der Funktion zur Übersetzung des vollständigen PDF-Texts; (2) Neues Feature zum Wechseln der Position des Eingabebereichs; (3) Hinzufügen der Option für eine vertikale Ausrichtung; (4) Verbesserung der Multithreading-Funktionen von Plugins.
|
|
||||||
- Version 2.3: Verbesserte Multithreading-Interaktivität
|
|
||||||
- Version 2.2: Funktionsplugins können heiß neu geladen werden
|
|
||||||
- Version 2.1: Faltbare Layouts
|
|
||||||
- Version 2.0: Einführung modularer Funktionsplugins
|
|
||||||
- Version 1.0: Grundfunktionen
|
|
||||||
|
|
||||||
Entwickler-QQ-Gruppe von GPT Academic: `610599535`
|
|
||||||
|
|
||||||
- Bekannte Probleme
|
|
||||||
- Einige Browserübersetzungsplugins beeinflussen die Frontend-Ausführung dieser Software
|
|
||||||
- Die offizielle Version von Gradio hat derzeit viele Kompatibilitätsprobleme. Installieren Sie Gradio daher unbedingt über `requirement.txt`.
|
|
||||||
|
|
||||||
### III: Themen
|
|
||||||
Sie können das Theme ändern, indem Sie die Option `THEME` (config.py) ändern.
|
|
||||||
1. `Chuanhu-Small-and-Beautiful` [Link](https://github.com/GaiZhenbiao/ChuanhuChatGPT/)
|
|
||||||
|
|
||||||
|
|
||||||
### IV: Entwicklungszweige dieses Projekts
|
|
||||||
|
|
||||||
1. `master` Branch: Hauptzweig, stabile Version
|
|
||||||
2. `frontier` Branch: Entwicklungsbranch, Testversion
|
|
||||||
|
|
||||||
|
|
||||||
### V: Referenzen und Lernen
|
|
||||||
|
|
||||||
```
|
|
||||||
Der Code basiert auf dem Design anderer herausragender Projekte. Die Reihenfolge ist beliebig:
|
|
||||||
|
|
||||||
# ChatGLM2-6B von Tsinghua:
|
|
||||||
https://github.com/THUDM/ChatGLM2-6B
|
|
||||||
|
|
||||||
# JittorLLMs von Tsinghua:
|
|
||||||
https://github.com/Jittor/JittorLLMs
|
|
||||||
|
|
||||||
# ChatPaper:
|
|
||||||
https://github.com/kaixindelele/ChatPaper
|
|
||||||
|
|
||||||
# Edge-GPT:
|
|
||||||
https://github.com/acheong08/EdgeGPT
|
|
||||||
|
|
||||||
# ChuanhuChatGPT:
|
|
||||||
https://github.com/GaiZhenbiao/ChuanhuChatGPT
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
# Oobabooga One-Click-Installations:
|
|
||||||
https://github.com/oobabooga/one-click-installers
|
|
||||||
|
|
||||||
# Weitere:
|
|
||||||
https://github.com/gradio-app/gradio
|
|
||||||
https://github.com/fghrsh/live2d_demo
|
|
||||||
@@ -1,360 +0,0 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
> **Nota**
|
|
||||||
>
|
|
||||||
> Questo README è stato tradotto da GPT (implementato da un plugin di questo progetto) e non è al 100% affidabile, per favore valuta attentamente i risultati della traduzione.
|
|
||||||
>
|
|
||||||
> 2023.11.7: Quando installi le dipendenze, seleziona le versioni **specificate** nel file `requirements.txt`. Comando di installazione: `pip install -r requirements.txt`.
|
|
||||||
|
|
||||||
|
|
||||||
# <div align=center><img src="logo.png" width="40"> GPT Ottimizzazione Accademica (GPT Academic)</div>
|
|
||||||
|
|
||||||
**Se ti piace questo progetto, per favore dagli una stella; se hai idee o plugin utili, fai una pull request!**
|
|
||||||
|
|
||||||
Se ti piace questo progetto, dagli una stella.
|
|
||||||
Per tradurre questo progetto in qualsiasi lingua con GPT, leggi ed esegui [`multi_language.py`](multi_language.py) (sperimentale).
|
|
||||||
|
|
||||||
> **Nota**
|
|
||||||
>
|
|
||||||
> 1. Fai attenzione che solo i plugin (pulsanti) **evidenziati** supportano la lettura dei file, alcuni plugin si trovano nel **menu a tendina** nell'area dei plugin. Inoltre, accogliamo e gestiamo con **massima priorità** qualsiasi nuovo plugin attraverso pull request.
|
|
||||||
>
|
|
||||||
> 2. Le funzioni di ogni file in questo progetto sono descritte in dettaglio nel [rapporto di traduzione automatica del progetto `self_analysis.md`](https://github.com/binary-husky/gpt_academic/wiki/GPT‐Academic项目自译解报告). Con l'iterazione della versione, puoi anche fare clic sui plugin delle funzioni rilevanti in qualsiasi momento per richiamare GPT e rigenerare il rapporto di auto-analisi del progetto. Domande frequenti [`wiki`](https://github.com/binary-husky/gpt_academic/wiki) | [Metodo di installazione standard](#installazione) | [Script di installazione one-click](https://github.com/binary-husky/gpt_academic/releases) | [Configurazione](https://github.com/binary-husky/gpt_academic/wiki/项目配置说明)。
|
|
||||||
>
|
|
||||||
> 3. Questo progetto è compatibile e incoraggia l'uso di modelli di linguaggio di grandi dimensioni nazionali, come ChatGLM. Supporto per la coesistenza di più chiavi API, puoi compilare nel file di configurazione come `API_KEY="openai-key1,openai-key2,azure-key3,api2d-key4"`. Quando è necessario sostituire temporaneamente `API_KEY`, inserisci temporaneamente `API_KEY` nell'area di input e premi Invio per confermare.
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
<div align="center">
|
|
||||||
|
|
||||||
Funzionalità (⭐ = Nuove funzionalità recenti) | Descrizione
|
|
||||||
--- | ---
|
|
||||||
⭐[Integrazione di nuovi modelli](https://github.com/binary-husky/gpt_academic/wiki/%E5%A6%82%E4%BD%95%E5%88%87%E6%8D%A2%E6%A8%A1%E5%9E%8B)！ | Baidu [Qianfan](https://cloud.baidu.com/doc/WENXINWORKSHOP/s/Nlks5zkzu) e [Wenxin](https://cloud.baidu.com/doc/GUIDE/5268.9) Intelligence, [Tongyi Qianwen](https://modelscope.cn/models/qwen/Qwen-7B-Chat/summary), Shanghai AI-Lab [bookbrain](https://github.com/InternLM/InternLM), Xunfei [Xinghuo](https://xinghuo.xfyun.cn/), [LLaMa2](https://huggingface.co/meta-llama/Llama-2-7b-chat-hf), Zhipu API, DALLE3
|
|
||||||
Revisione, traduzione, spiegazione del codice | Revisione, traduzione, ricerca errori grammaticali nei documenti e spiegazione del codice con un clic
|
|
||||||
[Tasti di scelta rapida personalizzati](https://www.bilibili.com/video/BV14s4y1E7jN) | Supporta tasti di scelta rapida personalizzati
|
|
||||||
Design modulare | Supporto per plugin personalizzati potenti, i plugin supportano l'[aggiornamento in tempo reale](https://github.com/binary-husky/gpt_academic/wiki/%E5%87%BD%E6%95%B0%E6%8F%92%E4%BB%B6%E6%8C%87%E5%8D%97)
|
|
||||||
[Analisi del codice](https://www.bilibili.com/video/BV1cj411A7VW) | [Plugin] Un clic per analizzare alberi di progetti Python/C/C++/Java/Lua/... o [autoanalisi](https://www.bilibili.com/video/BV1cj411A7VW)
|
|
||||||
Lettura di documenti, traduzione di documenti | [Plugin] Un clic per interpretare documenti completi in latex/pdf e generare un riassunto
|
|
||||||
Traduzione completa di testi in Latex, revisione completa di testi in Latex | [Plugin] Un clic per tradurre o correggere documenti in latex
|
|
||||||
Generazione automatica di commenti in batch | [Plugin] Un clic per generare commenti di funzione in batch
|
|
||||||
Traduzione [cinese-inglese](https://www.bilibili.com/video/BV1yo4y157jV/) in Markdown | [Plugin] Hai visto sopra i README in 5 lingue diverse ([Inglese](https://github.com/binary-husky/gpt_academic/blob/master/docs/README_EN.md))?
|
|
||||||
Generazione di rapporti di analisi chat | [Plugin] Genera automaticamente un rapporto di sintesi dopo l'esecuzione
|
|
||||||
Funzionalità di traduzione di testo completo in PDF | [Plugin] Estrai il titolo e il riassunto dei documenti PDF e traduci tutto il testo (multithreading)
|
|
||||||
Aiutante per Arxiv | [Plugin] Inserisci l'URL dell'articolo Arxiv per tradurre riassunto e scaricare PDF in un clic
|
|
||||||
Controllo completo dei documenti in Latex | [Plugin] Rileva errori grammaticali e ortografici nei documenti in Latex simile a Grammarly + Scarica un PDF per il confronto
|
|
||||||
Assistente per Google Scholar | [Plugin] Dato qualsiasi URL della pagina di ricerca di Google Scholar, fai scrivere da GPT gli *articoli correlati* per te
|
|
||||||
Concentrazione delle informazioni di Internet + GPT | [Plugin] [Recupera informazioni da Internet](https://www.bilibili.com/video/BV1om4y127ck) utilizzando GPT per rispondere alle domande e rendi le informazioni sempre aggiornate
|
|
||||||
⭐Traduzione accurata di articoli Arxiv ([Docker](https://github.com/binary-husky/gpt_academic/pkgs/container/gpt_academic_with_latex)) | [Plugin] [Traduci articoli Arxiv ad alta qualità](https://www.bilibili.com/video/BV1dz4y1v77A/) con un clic, lo strumento di traduzione degli articoli migliore al mondo al momento
|
|
||||||
⭐[Inserimento della conversazione vocale in tempo reale](https://github.com/binary-husky/gpt_academic/blob/master/docs/use_audio.md) | [Plugin] [Ascolta l'audio](https://www.bilibili.com/video/BV1AV4y187Uy/) in modo asincrono, taglia automaticamente le frasi e trova automaticamente il momento giusto per rispondere
|
|
||||||
Visualizzazione di formule, immagini, tabelle | Mostra contemporaneamente formule in formato tex e renderizzato, supporta formule e evidenziazione del codice
|
|
||||||
⭐Plugin multi-agente AutoGen | [Plugin] Esplora le possibilità dell'emergenza intelligence multi-agente con l'aiuto di Microsoft AutoGen!
|
|
||||||
Attiva il tema scuro [qui](https://github.com/binary-husky/gpt_academic/issues/173) | Aggiungi ```/?__theme=dark``` alla fine dell'URL del browser per passare al tema scuro
|
|
||||||
Supporto di più modelli LLM | Essere servito contemporaneamente da GPT3.5, GPT4, [ChatGLM2 di Tsinghua](https://github.com/THUDM/ChatGLM2-6B), [MOSS di Fudan](https://github.com/OpenLMLab/MOSS)
|
|
||||||
⭐Modello di fine-tuning ChatGLM2 | Supporto per l'importazione del modello di fine-tuning di ChatGLM2, fornendo plug-in di assistenza per il fine tuning di ChatGLM2
|
|
||||||
Più supporto per modelli LLM, supporto del [deploy di Huggingface](https://huggingface.co/spaces/qingxu98/gpt-academic) | Aggiungi interfaccia Newbing (Bing Translator), introduce il supporto di [JittorLLMs](https://github.com/Jittor/JittorLLMs) di Tsinghua, supporto per [LLaMA](https://github.com/facebookresearch/llama) e [Panguα](https://openi.org.cn/pangu/)
|
|
||||||
⭐Pacchetto pip [void-terminal](https://github.com/binary-husky/void-terminal) | Fornisce funzionalità di tutti i plugin di questo progetto direttamente in Python senza GUI (in sviluppo)
|
|
||||||
⭐Plugin terminale virtuale | [Plugin] Richiama altri plugin di questo progetto utilizzando linguaggio naturale
|
|
||||||
Altre nuove funzionalità (come la generazione di immagini) ... | Vedi alla fine di questo documento ...
|
|
||||||
|
|
||||||
</div>
|
|
||||||
|
|
||||||
|
|
||||||
- Nuovo layout (modifica l'opzione LAYOUT in `config.py` per passare tra "layout sinistra / destra" e "layout sopra / sotto")
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://github.com/binary-husky/gpt_academic/assets/96192199/d81137c3-affd-4cd1-bb5e-b15610389762" width="700" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
|
|
||||||
- Tutti i pulsanti vengono generati dinamicamente leggendo `functional.py`, puoi aggiungere liberamente funzionalità personalizzate, liberando la clipboard
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/231975334-b4788e91-4887-412f-8b43-2b9c5f41d248.gif" width="700" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
- Revisione / correzione
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/231980294-f374bdcb-3309-4560-b424-38ef39f04ebd.gif" width="700" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
- Se l'output contiene formule, saranno visualizzate sia in formato tex che in formato renderizzato per facilitarne la copia e la lettura.
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/230598842-1d7fcddd-815d-40ee-af60-baf488a199df.png" width="700" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
- Non hai voglia di guardare il codice del progetto? Mostralo direttamente al chatgpt in bocca.
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/226935232-6b6a73ce-8900-4aee-93f9-733c7e6fef53.png" width="700" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
- Chiamate miste di modelli di grandi dimensioni (ChatGLM + OpenAI-GPT3.5 + [API2D](https://api2d.com/)-GPT4)
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/232537274-deca0563-7aa6-4b5d-94a2-b7c453c47794.png" width="700" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
# Installazione
|
|
||||||
### Metodo di installazione I: Esegui direttamente (Windows, Linux o MacOS)
|
|
||||||
|
|
||||||
1. Scarica il progetto
|
|
||||||
```sh
|
|
||||||
git clone --depth=1 https://github.com/binary-husky/gpt_academic.git
|
|
||||||
cd gpt_academic
|
|
||||||
```
|
|
||||||
|
|
||||||
2. Configura l'API_KEY
|
|
||||||
|
|
||||||
Nel file `config.py`, configura l'API KEY e altre impostazioni, [clicca qui per vedere come configurare l'API in ambienti di rete speciali](https://github.com/binary-husky/gpt_academic/issues/1) . [Pagina Wiki](https://github.com/binary-husky/gpt_academic/wiki/项目配置说明).
|
|
||||||
|
|
||||||
「 Il programma controllerà prima se esiste un file di configurazione privata chiamato `config_private.py` e utilizzerà le configurazioni in esso contenute per sovrascrivere le configurazioni con lo stesso nome in `config.py`. Se comprendi questa logica di lettura, ti consigliamo vivamente di creare un nuovo file di configurazione chiamato `config_private.py` accanto a `config.py` e spostare (copiare) le configurazioni da `config.py` a `config_private.py` (basta copiare le voci di configurazione che hai modificato). 」
|
|
||||||
|
|
||||||
「 Supporta la configurazione del progetto tramite `variabili d'ambiente`, il formato di scrittura delle variabili d'ambiente è descritto nel file `docker-compose.yml` o nella nostra [pagina Wiki](https://github.com/binary-husky/gpt_academic/wiki/项目配置说明) priorità di lettura della configurazione: `variabili d'ambiente` > `config_private.py` > `config.py`. 」
|
|
||||||
|
|
||||||
3. Installa le dipendenze
|
|
||||||
```sh
|
|
||||||
# (Scelta I: Se familiarizzato con python, python>=3.9) Nota: Usa il repository delle fonti ufficiale di pip o Ali pip per temporaneamente cambiare la fonte: python -m pip install -r requirements.txt -i https://mirrors.aliyun.com/pypi/simple/
|
|
||||||
python -m pip install -r requirements.txt
|
|
||||||
|
|
||||||
# (Scelta II: Usa Anaconda) Anche in questo caso, i passaggi sono simili (https://www.bilibili.com/video/BV1rc411W7Dr):
|
|
||||||
conda create -n gptac_venv python=3.11 # Crea l'ambiente anaconda
|
|
||||||
conda activate gptac_venv # Attiva l'ambiente anaconda
|
|
||||||
python -m pip install -r requirements.txt # Questo passaggio è identico alla procedura di installazione con pip
|
|
||||||
```
|
|
||||||
|
|
||||||
|
|
||||||
<details><summary>Se desideri utilizzare il backend di ChatGLM2 di Tsinghua/Fudan MOSS/RWKV, fai clic per espandere</summary>
|
|
||||||
<p>
|
|
||||||
|
|
||||||
[Optional] Se desideri utilizzare ChatGLM2 di Tsinghua/Fudan MOSS come backend, è necessario installare ulteriori dipendenze (Requisiti: conoscenza di Python + esperienza con Pytorch + hardware potente):
|
|
||||||
```sh
|
|
||||||
# [Optional Step I] Supporto per ChatGLM2 di Tsinghua. Note di ChatGLM di Tsinghua: Se si verifica l'errore "Call ChatGLM fail non può caricare i parametri di ChatGLM", fare riferimento a quanto segue: 1: L'installazione predefinita è la versione torch+cpu, per usare cuda è necessario disinstallare torch ed installare nuovamente la versione con torch+cuda; 2: Se il modello non può essere caricato a causa di una configurazione insufficiente, è possibile modificare la precisione del modello in request_llm/bridge_chatglm.py, sostituendo AutoTokenizer.from_pretrained("THUDM/chatglm-6b", trust_remote_code=True) con AutoTokenizer.from_pretrained("THUDM/chatglm-6b-int4", trust_remote_code=True)
|
|
||||||
python -m pip install -r request_llms/requirements_chatglm.txt
|
|
||||||
|
|
||||||
# [Optional Step II] Supporto per Fudan MOSS
|
|
||||||
python -m pip install -r request_llms/requirements_moss.txt
|
|
||||||
git clone --depth=1 https://github.com/OpenLMLab/MOSS.git request_llms/moss # Attenzione: eseguire questo comando nella directory principale del progetto
|
|
||||||
|
|
||||||
# [Optional Step III] Supporto per RWKV Runner
|
|
||||||
Consulta il Wiki: https://github.com/binary-husky/gpt_academic/wiki/%E9%80%82%E9%85%8DRWKV-Runner
|
|
||||||
|
|
||||||
# [Optional Step IV] Assicurati che il file di configurazione config.py includa i modelli desiderati. Di seguito sono elencati i modelli attualmente supportati (gli llm di jittorllms supportano solo la soluzione Docker):
|
|
||||||
AVAIL_LLM_MODELS = ["gpt-3.5-turbo", "api2d-gpt-3.5-turbo", "gpt-4", "api2d-gpt-4", "chatglm", "moss", "jittorllms_rwkv", "jittorllms_pangualpha", "jittorllms_llama"]
|
|
||||||
```
|
|
||||||
|
|
||||||
</p>
|
|
||||||
</details>
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
4. Esegui
|
|
||||||
```sh
|
|
||||||
python main.py
|
|
||||||
```
|
|
||||||
|
|
||||||
### Metodo di installazione II: Utilizzo di Docker
|
|
||||||
|
|
||||||
0. Installa tutte le funzionalità del progetto (Questo è un'immagine di grandi dimensioni che include cuda e latex. Potrebbe non essere adatta se hai una connessione lenta o uno spazio su disco limitato)
|
|
||||||
[](https://github.com/binary-husky/gpt_academic/actions/workflows/build-with-all-capacity.yml)
|
|
||||||
|
|
||||||
``` sh
|
|
||||||
# Modifica il file docker-compose.yml: mantieni solo la configurazione 0 e rimuovi le altre configurazioni. Avvia il seguente comando:
|
|
||||||
docker-compose up
|
|
||||||
```
|
|
||||||
|
|
||||||
1. ChatGPT + Wenxin Yiyu (Poem) + Spark, solo modelli online (Consigliato per la maggior parte delle persone)
|
|
||||||
[](https://github.com/binary-husky/gpt_academic/actions/workflows/build-without-local-llms.yml)
|
|
||||||
[](https://github.com/binary-husky/gpt_academic/actions/workflows/build-with-latex.yml)
|
|
||||||
[](https://github.com/binary-husky/gpt_academic/actions/workflows/build-with-audio-assistant.yml)
|
|
||||||
|
|
||||||
``` sh
|
|
||||||
# Modifica il file docker-compose.yml: mantieni solo la configurazione 1 e rimuovi le altre configurazioni. Avvia il seguente comando:
|
|
||||||
docker-compose up
|
|
||||||
```
|
|
||||||
|
|
||||||
P.S. Se hai bisogno del plugin LaTeX, consulta la pagina Wiki. In alternativa, puoi utilizzare le configurazioni 4 o 0 direttamente per ottenere questa funzionalità.
|
|
||||||
|
|
||||||
2. ChatGPT + ChatGLM2 + MOSS + LLAMA2 + Tongyi Q&W (Richiede conoscenze su Nvidia Docker)
|
|
||||||
[](https://github.com/binary-husky/gpt_academic/actions/workflows/build-with-chatglm.yml)
|
|
||||||
|
|
||||||
``` sh
|
|
||||||
# Modifica il file docker-compose.yml: mantieni solo la configurazione 2 e rimuovi le altre configurazioni. Avvia il seguente comando:
|
|
||||||
docker-compose up
|
|
||||||
```
|
|
||||||
|
|
||||||
|
|
||||||
### Metodo di installazione III: Altre opzioni di distribuzione
|
|
||||||
1. **Script di esecuzione con un clic per Windows**.
|
|
||||||
Se non conosci affatto l'ambiente python in Windows, puoi scaricare uno script di esecuzione con un clic dalla sezione [Release](https://github.com/binary-husky/gpt_academic/releases) per installare la versione che non richiede modelli locali.
|
|
||||||
Lo script è stato fornito da [oobabooga](https://github.com/oobabooga/one-click-installers).
|
|
||||||
|
|
||||||
2. Utilizzo di API di terze parti, Azure, Wenxin Yiyu (Poem), Xinghuo, ecc. vedi [pagina Wiki](https://github.com/binary-husky/gpt_academic/wiki/项目配置说明)
|
|
||||||
|
|
||||||
3. Guida all'installazione del server cloud remoto.
|
|
||||||
Visita la [pagina Wiki sull'installazione del server cloud remoto](https://github.com/binary-husky/gpt_academic/wiki/云服务器远程部署指南).
|
|
||||||
|
|
||||||
4. Altre nuove piattaforme o metodi di distribuzione:
|
|
||||||
- Uso di Sealos per il [deployment con un clic](https://github.com/binary-husky/gpt_academic/issues/993).
|
|
||||||
- Uso di WSL2 (Windows Subsystem for Linux). Vedi [Guida all'installazione](https://github.com/binary-husky/gpt_academic/wiki/使用WSL2（Windows-Subsystem-for-Linux-子系统）部署) per maggiori informazioni.
|
|
||||||
- Funzionamento su un sotto-percorso URL (`http://localhost/subpath`). Vedi [istruzioni FastAPI](docs/WithFastapi.md) per maggiori dettagli.
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
# Utilizzo avanzato
|
|
||||||
### I: Personalizzare nuovi pulsanti rapidi (tasti di scelta rapida accademici)
|
|
||||||
Apri `core_functional.py` con qualsiasi editor di testo e aggiungi le seguenti voci, quindi riavvia il programma. (Se il pulsante esiste già, sia il prefisso che il suffisso possono essere modificati a caldo senza la necessità di riavviare il programma.)
|
|
||||||
Ad esempio,
|
|
||||||
```
|
|
||||||
"Traduzione avanzata Cinese-Inglese": {
|
|
||||||
# Prefisso, sarà aggiunto prima del tuo input. Ad esempio, utilizzato per descrivere la tua richiesta, come traduzione, spiegazione del codice, rifinitura, ecc.
|
|
||||||
"Prefisso": "Si prega di tradurre il seguente testo in cinese e fornire spiegazione per i termini tecnici utilizzati, utilizzando una tabella in markdown uno per uno:\n\n",
|
|
||||||
|
|
||||||
# Suffisso, sarà aggiunto dopo il tuo input. Ad esempio, in combinazione con il prefisso, puoi circondare il tuo input con virgolette.
|
|
||||||
"Suffisso": "",
|
|
||||||
},
|
|
||||||
```
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/226899272-477c2134-ed71-4326-810c-29891fe4a508.png" width="500" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
### II: Plugin di funzioni personalizzate
|
|
||||||
Scrivi potentissimi plugin di funzioni per eseguire qualsiasi compito che desideri, sia che tu lo pensi o meno.
|
|
||||||
La scrittura di plugin per questo progetto è facile e richiede solo conoscenze di base di Python. Puoi seguire il [Guida ai Plugin di Funzione](https://github.com/binary-husky/gpt_academic/wiki/%E5%87%BD%E6%95%B0%E6%8F%92%E4%BB%B6%E6%8C%87%E5%8D%97) per maggiori dettagli.
|
|
||||||
|
|
||||||
|
|
||||||
# Aggiornamenti
|
|
||||||
### I: Aggiornamenti
|
|
||||||
|
|
||||||
1. Funzionalità di salvataggio della conversazione. Chiamare `Salva la conversazione corrente` nell'area del plugin per salvare la conversazione corrente come un file html leggibile e ripristinabile.
|
|
||||||
Inoltre, nella stessa area del plugin (menu a tendina) chiamare `Carica la cronologia della conversazione` per ripristinare una conversazione precedente.
|
|
||||||
Suggerimento: fare clic su `Carica la cronologia della conversazione` senza specificare un file per visualizzare la tua cronologia di archiviazione HTML.
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/235222390-24a9acc0-680f-49f5-bc81-2f3161f1e049.png" width="500" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
2. ⭐ Funzionalità di traduzione articoli Latex/Arxiv ⭐
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://github.com/binary-husky/gpt_academic/assets/96192199/002a1a75-ace0-4e6a-94e2-ec1406a746f1" height="250" > ===>
|
|
||||||
<img src="https://github.com/binary-husky/gpt_academic/assets/96192199/9fdcc391-f823-464f-9322-f8719677043b" height="250" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
3. Terminale vuoto (Comprensione dell'intento dell'utente dai testi liberi + Chiamata automatica di altri plugin)
|
|
||||||
|
|
||||||
- Passaggio 1: Digitare "Chiamare il plugin per tradurre un documento PDF, l'indirizzo è https://openreview.net/pdf?id=rJl0r3R9KX"
|
|
||||||
- Passaggio 2: Fare clic su "Terminale vuoto"
|
|
||||||
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://github.com/binary-husky/gpt_academic/assets/96192199/66f1b044-e9ff-4eed-9126-5d4f3668f1ed" width="500" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
4. Design modulare, interfacce semplici che supportano funzionalità potenti
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/229288270-093643c1-0018-487a-81e6-1d7809b6e90f.png" height="400" >
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/227504931-19955f78-45cd-4d1c-adac-e71e50957915.png" height="400" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
5. Traduzione e interpretazione di altri progetti open source
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/226935232-6b6a73ce-8900-4aee-93f9-733c7e6fef53.png" height="250" >
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/226969067-968a27c1-1b9c-486b-8b81-ab2de8d3f88a.png" height="250" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
6. Funzionalità leggera per [live2d](https://github.com/fghrsh/live2d_demo) (disabilitata per impostazione predefinita, richiede modifica di `config.py`)
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/236432361-67739153-73e8-43fe-8111-b61296edabd9.png" width="500" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
7. Generazione di immagini di OpenAI
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://github.com/binary-husky/gpt_academic/assets/96192199/bc7ab234-ad90-48a0-8d62-f703d9e74665" width="500" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
8. Elaborazione e riepilogo audio di OpenAI
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://github.com/binary-husky/gpt_academic/assets/96192199/709ccf95-3aee-498a-934a-e1c22d3d5d5b" width="500" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
9. Correzione totale del testo di Latex
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://github.com/binary-husky/gpt_academic/assets/96192199/651ccd98-02c9-4464-91e1-77a6b7d1b033" height="200" > ===>
|
|
||||||
<img src="https://github.com/binary-husky/gpt_academic/assets/96192199/476f66d9-7716-4537-b5c1-735372c25adb" height="200">
|
|
||||||
</div>
|
|
||||||
|
|
||||||
10. Cambio linguaggio e tema
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://github.com/binary-husky/gpt_academic/assets/96192199/b6799499-b6fb-4f0c-9c8e-1b441872f4e8" width="500" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
|
|
||||||
### II: Versioni:
|
|
||||||
- versione 3.70 (todo): Ottimizzazione della visualizzazione del tema AutoGen e sviluppo di una serie di plugin correlati.
|
|
||||||
- versione 3.60: Introduzione di AutoGen come fondamento per i plugin della nuova generazione.
|
|
||||||
- versione 3.57: Supporto per GLM3, StarFirev3, Wenxin-yiyanv4 e correzione di bug sulla concorrenza dell'uso di modelli locali.
|
|
||||||
- versione 3.56: Possibilità di aggiungere dinamicamente pulsanti per funzionalità di base e nuova pagina di riepilogo del PDF.
|
|
||||||
- versione 3.55: Ristrutturazione dell'interfaccia utente, introduzione di finestre fluttuanti e barre dei menu.
|
|
||||||
- versione 3.54: Nuovo interprete di codice dinamico (Code Interpreter) (da perfezionare).
|
|
||||||
- versione 3.53: Possibilità di selezionare dinamicamente diversi temi dell'interfaccia utente, miglioramento della stabilità e risoluzione dei conflitti tra utenti multipli.
|
|
||||||
- versione 3.50: Utilizzo del linguaggio naturale per chiamare tutte le funzioni dei plugin di questo progetto (Terminale vuoto), supporto per la classificazione dei plugin, miglioramento dell'interfaccia utente e design di nuovi temi.
|
|
||||||
- versione 3.49: Supporto per la piattaforma Baidu Qianfan e Wenxin-yiyan.
|
|
||||||
- versione 3.48: Supporto per Alibaba DAXI 所见即所答, Shanghai AI-Lab Shusheng, Xunfei StarFire.
|
|
||||||
- versione 3.46: Supporto per la chat vocale in tempo reale completamente automatica.
|
|
||||||
- versione 3.45: Supporto personalizzato per il micro-aggiustamento del modello ChatGLM2.
|
|
||||||
- versione 3.44: Supporto ufficiale per Azure, miglioramento dell'usabilità dell'interfaccia.
|
|
||||||
- versione 3.4: + Funzionalità di traduzione di documenti arXiv e correzione di documenti LaTeX.
|
|
||||||
- versione 3.3: + Funzionalità di sintesi delle informazioni su Internet.
|
|
||||||
- versione 3.2: Il plugin di funzione supporta più interfacce dei parametri (funzionalità di salvataggio della conversazione, interpretazione di codici in qualsiasi linguaggio contemporaneamente, interrogare qualsiasi combinazione di LLM).
|
|
||||||
- versione 3.1: Supporto per l'interrogazione simultanea di più modelli GPT! Supporto per api2d, equilibrio del carico con più apikey.
|
|
||||||
- versione 3.0: Supporto per chatglm e altri piccoli llm.
|
|
||||||
- versione 2.6: Rielaborazione della struttura del plugin, miglioramento dell'interattività, aggiunta di ulteriori plugin.
|
|
||||||
- versione 2.5: Aggiornamento automatico, risoluzione del problema della lunghezza eccessiva del testo durante il riepilogo di grandi blocchi di codice che supera i token.
|
|
||||||
- versione 2.4: (1) Nuova funzionalità di traduzione di documenti PDF; (2) Nuova funzionalità di scambio delle posizioni tra l'area di input (input area); (3) Nuova opzione di layout verticale; (4) Ottimizzazione del plugin a threading multiplo.
|
|
||||||
- versione 2.3: Miglioramento dell'interattività con threading multiplo.
|
|
||||||
- versione 2.2: Supporto per il plugin con ricarica a caldo.
|
|
||||||
- versione 2.1: Layout pieghevole.
|
|
||||||
- versione 2.0: Introduzione di plugin modulari.
|
|
||||||
- versione 1.0: Funzioni di base
|
|
||||||
|
|
||||||
GPT Academic Developer QQ Group: `610599535`
|
|
||||||
|
|
||||||
- Problemi noti
|
|
||||||
- Alcuni plug-in di traduzione del browser possono interferire con il funzionamento del frontend di questo software
|
|
||||||
- L'app Gradio ufficiale ha molti bug di compatibilità, si consiglia di installare Gradio tramite `requirement.txt`
|
|
||||||
|
|
||||||
### III: Temi
|
|
||||||
Il tema può essere modificato modificando l'opzione `THEME` (config.py)
|
|
||||||
1. `Chuanhu-Small-and-Beautiful` [Website](https://github.com/GaiZhenbiao/ChuanhuChatGPT/)
|
|
||||||
|
|
||||||
|
|
||||||
### IV: Branch di Sviluppo di questo progetto
|
|
||||||
|
|
||||||
1. `master` branch: branch principale, versione stabile
|
|
||||||
2. `frontier` branch: branch di sviluppo, versione di test
|
|
||||||
|
|
||||||
|
|
||||||
### V: Riferimenti e Risorse di Apprendimento
|
|
||||||
|
|
||||||
```
|
|
||||||
Nel codice sono state utilizzate diverse idee dagli altri progetti, senza un ordine specifico:
|
|
||||||
|
|
||||||
# ChatGLM2-6B di Tsinghua:
|
|
||||||
https://github.com/THUDM/ChatGLM2-6B
|
|
||||||
|
|
||||||
# JittorLLMs di Tsinghua:
|
|
||||||
https://github.com/Jittor/JittorLLMs
|
|
||||||
|
|
||||||
# ChatPaper:
|
|
||||||
https://github.com/kaixindelele/ChatPaper
|
|
||||||
|
|
||||||
# Edge-GPT:
|
|
||||||
https://github.com/acheong08/EdgeGPT
|
|
||||||
|
|
||||||
# ChuanhuChatGPT:
|
|
||||||
https://github.com/GaiZhenbiao/ChuanhuChatGPT
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
# Installazione con un solo clic di Oobabooga:
|
|
||||||
https://github.com/oobabooga/one-click-installers
|
|
||||||
|
|
||||||
# Altre risorse:
|
|
||||||
https://github.com/gradio-app/gradio
|
|
||||||
https://github.com/fghrsh/live2d_demo
|
|
||||||
@@ -1,344 +0,0 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
> **注意**
|
|
||||||
>
|
|
||||||
> 此READMEはGPTによる翻訳で生成されました（このプロジェクトのプラグインによって実装されています）、翻訳結果は100%正確ではないため、注意してください。
|
|
||||||
>
|
|
||||||
> 2023年11月7日: 依存関係をインストールする際は、`requirements.txt`で**指定されたバージョン**を選択してください。 インストールコマンド: `pip install -r requirements.txt`。
|
|
||||||
|
|
||||||
|
|
||||||
# <div align=center><img src="logo.png" width="40"> GPT 学術最適化 (GPT Academic)</div>
|
|
||||||
|
|
||||||
**このプロジェクトが気に入った場合は、Starを付けてください。また、便利なショートカットキーまたはプラグインを作成した場合は、プルリクエストを歓迎します！**
|
|
||||||
GPTを使用してこのプロジェクトを任意の言語に翻訳するには、[`multi_language.py`](multi_language.py)を読み込んで実行します（実験的な機能）。
|
|
||||||
|
|
||||||
> **注意**
|
|
||||||
>
|
|
||||||
> 1. **強調された** プラグイン（ボタン）のみがファイルを読み込むことができることに注意してください。一部のプラグインは、プラグインエリアのドロップダウンメニューにあります。また、新しいプラグインのPRを歓迎し、最優先で対応します。
|
|
||||||
>
|
|
||||||
> 2. このプロジェクトの各ファイルの機能は、[自己分析レポート`self_analysis.md`](https://github.com/binary-husky/gpt_academic/wiki/GPT‐Academic%E9%A1%B9%E7%9B%AE%E8%87%AA%E8%AF%91%E8%A7%A3%E5%A0%82)で詳しく説明されています。バージョンが進化するにつれて、関連する関数プラグインをクリックして、プロジェクトの自己分析レポートをGPTで再生成することもできます。よくある質問については、[`wiki`](https://github.com/binary-husky/gpt_academic/wiki)をご覧ください。[標準的なインストール方法](#installation) | [ワンクリックインストールスクリプト](https://github.com/binary-husky/gpt_academic/releases) | [構成の説明](https://github.com/binary-husky/gpt_academic/wiki/Project-Configuration-Explain)。
|
|
||||||
>
|
|
||||||
> 3. このプロジェクトは、[ChatGLM](https://www.chatglm.dev/)などの中国製の大規模言語モデルも互換性があり、試してみることを推奨しています。複数のAPIキーを共存させることができ、設定ファイルに`API_KEY="openai-key1,openai-key2,azure-key3,api2d-key4"`のように記入できます。`API_KEY`を一時的に変更する必要がある場合は、入力エリアに一時的な`API_KEY`を入力し、Enterキーを押して提出すると有効になります。
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
<div align="center">
|
|
||||||
|
|
||||||
機能（⭐= 最近追加された機能） | 説明
|
|
||||||
--- | ---
|
|
||||||
⭐[新しいモデルの追加](https://github.com/binary-husky/gpt_academic/wiki/%E5%A6%82%E4%BD%95%E5%88%87%E6%8D%A2%E6%A8%A1%E5%9E%8B)！ | Baidu [Qianfan](https://cloud.baidu.com/doc/WENXINWORKSHOP/s/Nlks5zkzu)とWenxin Yiyu, [Tongyi Qianwen](https://modelscope.cn/models/qwen/Qwen-7B-Chat/summary), Shanghai AI-Lab [Shusheng](https://github.com/InternLM/InternLM), Xunfei [Xinghuo](https://xinghuo.xfyun.cn/), [LLaMa2](https://huggingface.co/meta-llama/Llama-2-7b-chat-hf), Zhantu API, DALLE3
|
|
||||||
校正、翻訳、コード解説 | 一括校正、翻訳、論文の文法エラーの検索、コードの解説
|
|
||||||
[カスタムショートカットキー](https://www.bilibili.com/video/BV14s4y1E7jN) | カスタムショートカットキーのサポート
|
|
||||||
モジュール化された設計 | カスタムでパワフルな[プラグイン](https://github.com/binary-husky/gpt_academic/tree/master/crazy_functions)のサポート、プラグインの[ホットリロード](https://github.com/binary-husky/gpt_academic/wiki/%E5%87%BD%E6%95%B0%E6%8F%92%E4%BB%B6%E6%8C%87%E5%8D%97)
|
|
||||||
[プログラム解析](https://www.bilibili.com/video/BV1cj411A7VW) | [プラグイン] Python/C/C++/Java/Lua/...のプロジェクトツリーを簡単に解析するか、[自己解析](https://www.bilibili.com/video/BV1cj411A7VW)
|
|
||||||
論文の読み込み、[翻訳](https://www.bilibili.com/video/BV1KT411x7Wn) | [プラグイン] LaTeX/PDFの論文全文を翻訳して要約を作成する
|
|
||||||
LaTeX全文の[翻訳](https://www.bilibili.com/video/BV1nk4y1Y7Js/)、[校正](https://www.bilibili.com/video/BV1FT411H7c5/) | [プラグイン] LaTeX論文を翻訳や校正する
|
|
||||||
一括コメント生成 | [プラグイン] 関数コメントを一括生成する
|
|
||||||
Markdownの[日英翻訳](https://www.bilibili.com/video/BV1yo4y157jV/) | [プラグイン] 5つの言語（[英語](https://github.com/binary-husky/gpt_academic/blob/master/docs/README_EN.md)など）のREADMEをご覧になりましたか？
|
|
||||||
チャット分析レポートの生成 | [プラグイン] 実行後にサマリーレポートを自動生成する
|
|
||||||
[PDF論文全文の翻訳機能](https://www.bilibili.com/video/BV1KT411x7Wn) | [プラグイン] PDF論文のタイトルと要約を抽出し、全文を翻訳する（マルチスレッド）
|
|
||||||
[Arxivアシスタント](https://www.bilibili.com/video/BV1LM4y1279X) | [プラグイン] arxiv論文のURLを入力すると、要約を翻訳してPDFをダウンロードできます
|
|
||||||
LaTeX論文の一括校正 | [プラグイン] Grammarlyのように、LaTeX論文の文法とスペルを修正して対照PDFを出力する
|
|
||||||
[Google Scholar統合アシスタント](https://www.bilibili.com/video/BV19L411U7ia) | [プラグイン] 任意のGoogle Scholar検索ページのURLを指定して、関連資料をGPTに書かせることができます
|
|
||||||
インターネット情報の集約+GPT | [プラグイン] インターネットから情報を取得して質問に答え、情報が常に最新になるようにします
|
|
||||||
⭐Arxiv論文の詳細な翻訳 ([Docker](https://github.com/binary-husky/gpt_academic/pkgs/container/gpt_academic_with_latex)) | [プラグイン] arxiv論文を超高品質で翻訳します。最高の論文翻訳ツールです
|
|
||||||
⭐[リアルタイム音声入力](https://github.com/binary-husky/gpt_academic/blob/master/docs/use_audio.md) | [プラグイン] 非同期[音声をリッスン(https://www.bilibili.com/video/BV1AV4y187Uy/)し、自動で文章を区切り、回答のタイミングを自動で探します
|
|
||||||
公式/画像/表の表示 | 公式の[tex形式とレンダリング形式](https://user-images.githubusercontent.com/96192199/230598842-1d7fcddd-815d-40ee-af60-baf488a199df.png)を同時に表示し、公式とコードのハイライトをサポートします
|
|
||||||
⭐AutoGenマルチエージェントプラグイン | [プラグイン] Microsoft AutoGenを利用して、マルチエージェントのインテリジェントなエマージェンスを探索します
|
|
||||||
ダーク[テーマ](https://github.com/binary-husky/gpt_academic/issues/173)を起動 | ブラウザのURLに```/?__theme=dark```を追加すると、ダークテーマに切り替えられます
|
|
||||||
[複数のLLMモデル](https://www.bilibili.com/video/BV1wT411p7yf)のサポート | GPT3.5、GPT4、[Tsinghua ChatGLM2](https://github.com/THUDM/ChatGLM2-6B)、[Fudan MOSS](https://github.com/OpenLMLab/MOSS)などを同時に使えるのは最高の感じですよね？
|
|
||||||
⭐ChatGLM2ファインチューニングモデル | ChatGLM2ファインチューニングモデルをロードして使用することができ、ChatGLM2ファインチューニングの補助プラグインが用意されています
|
|
||||||
さらなるLLMモデルの導入、[HuggingFaceデプロイのサポート](https://huggingface.co/spaces/qingxu98/gpt-academic) | Newbingインターフェース（新しいBing）の追加、Tsinghua [Jittorllms](https://github.com/Jittor/JittorLLMs)の導入、[LLaMA](https://github.com/facebookresearch/llama)および[盤古α](https://openi.org.cn/pangu/)のサポート
|
|
||||||
⭐[void-terminal](https://github.com/binary-husky/void-terminal) pipパッケージ | GUIから独立して、Pythonから直接このプロジェクトのすべての関数プラグインを呼び出せます（開発中）
|
|
||||||
⭐Void Terminalプラグイン | [プラグイン] 自然言語で、このプロジェクトの他のプラグインを直接実行します
|
|
||||||
その他の新機能の紹介（画像生成など）...... | 末尾をご覧ください ......
|
|
||||||
</div>
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
- もし出力に数式が含まれている場合、TeX形式とレンダリング形式の両方で表示されます。これにより、コピーと読み取りが容易になります。
|
|
||||||
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/230598842-1d7fcddd-815d-40ee-af60-baf488a199df.png" width="700" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
- プロジェクトのコードを見るのがめんどくさい？プロジェクト全体を`chatgpt`に広報口頭発表してもらえるよ
|
|
||||||
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/226935232-6b6a73ce-8900-4aee-93f9-733c7e6fef53.png" width="700" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
- 異なる言語モデルの組み合わせ呼び出し（ChatGLM + OpenAI-GPT3.5 + [API2D](https://api2d.com/)-GPT4）
|
|
||||||
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/232537274-deca0563-7aa6-4b5d-94a2-b7c453c47794.png" width="700" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
# インストール
|
|
||||||
### 方法I：直接実行（Windows、Linux、またはMacOS）
|
|
||||||
|
|
||||||
1. プロジェクトをダウンロード
|
|
||||||
```sh
|
|
||||||
git clone --depth=1 https://github.com/binary-husky/gpt_academic.git
|
|
||||||
cd gpt_academic
|
|
||||||
```
|
|
||||||
|
|
||||||
2. APIキーを設定する
|
|
||||||
|
|
||||||
`config.py`でAPIキーやその他の設定を設定します。特殊なネットワーク環境の設定方法については、[こちらをクリックして確認してください](https://github.com/binary-husky/gpt_academic/issues/1)。[Wikiページ](https://github.com/binary-husky/gpt_academic/wiki/Getting-Started)も参照してください。
|
|
||||||
|
|
||||||
「プログラムは、`config.py`と同じ場所にある`config_private.py`という名前のプライベート設定ファイルが存在するかどうかを優先的にチェックし、同じ名前の設定をコピーします。この読み込みロジックを理解できる場合、`config.py`の横に`config_private.py`という名前の新しい設定ファイルを作成し、`config.py`の設定を転送（コピー）することを強くお勧めします（変更した設定項目だけをコピーします）。」
|
|
||||||
|
|
||||||
「プロジェクトを環境変数で設定することもサポートしています。環境変数の書式は、`docker-compose.yml`ファイルや[Wikiページ](https://github.com/binary-husky/gpt_academic/wiki/Getting-Started)を参考にしてください。設定の優先度は、`環境変数` > `config_private.py` > `config.py`の順です。」
|
|
||||||
|
|
||||||
3. 依存関係をインストールする
|
|
||||||
```sh
|
|
||||||
# （オプションI：Pythonに詳しい場合、Python 3.9以上）注：公式のpipソースまたは阿里pipソースを使用し、一時的なソースの変更方法は、python -m pip install -r requirements.txt -i https://mirrors.aliyun.com/pypi/simple/です。
|
|
||||||
python -m pip install -r requirements.txt
|
|
||||||
|
|
||||||
# （オプションII：Anacondaを使用する場合）手順は同様です (https://www.bilibili.com/video/BV1rc411W7Dr)：
|
|
||||||
conda create -n gptac_venv python=3.11 # Anaconda環境を作成
|
|
||||||
conda activate gptac_venv # Anaconda環境をアクティベート
|
|
||||||
python -m pip install -r requirements.txt # この手順はpipのインストール手順と同じです
|
|
||||||
```
|
|
||||||
|
|
||||||
<details><summary>清華ChatGLM2/復旦MOSS/RWKVがバックエンドとしてサポートされている場合、ここをクリックして展開してください</summary>
|
|
||||||
<p>
|
|
||||||
|
|
||||||
【オプションステップ】 清華ChatGLM2/復旦MOSSをバックエンドとしてサポートする場合は、さらに追加の依存関係をインストールする必要があります（前提条件：Pythonに精通していて、PytorchとNVIDIA GPUを使用したことがあり、十分なコンピュータの構成を持っていること）：
|
|
||||||
|
|
||||||
```sh
|
|
||||||
# 【オプションステップI】 清華ChatGLM2のサポートを追加する。 清華ChatGLM2に関する注意点： "Call ChatGLM fail 不能正常加载ChatGLM的参数" のエラーが発生した場合、次の手順を参照してください。 1: デフォルトでインストールされているのはtorch+cpu版です。CUDAを使用するにはtorchをアンインストールしてtorch+cuda版を再インストールする必要があります。 2: モデルをロードできない場合は、request_llm/bridge_chatglm.pyのモデル精度を変更できます。AutoTokenizer.from_pretrained("THUDM/chatglm-6b", trust_remote_code=True)をAutoTokenizer.from_pretrained("THUDM/chatglm-6b-int4", trust_remote_code=True)に変更します。
|
|
||||||
python -m pip install -r request_llms/requirements_chatglm.txt
|
|
||||||
|
|
||||||
# 【オプションステップII】 復旦MOSSのサポートを追加する
|
|
||||||
python -m pip install -r request_llms/requirements_moss.txt
|
|
||||||
git clone --depth=1 https://github.com/OpenLMLab/MOSS.git request_llms/moss # このコマンドを実行するときは、プロジェクトのルートパスである必要があります。
|
|
||||||
|
|
||||||
# 【オプションステップIII】 RWKV Runnerのサポートを追加する
|
|
||||||
Wikiを参照してください: https://github.com/binary-husky/gpt_academic/wiki/%E9%80%82%E9%85%8DRWKV-Runner
|
|
||||||
|
|
||||||
# 【オプションステップIV】 config.py設定ファイルに、以下のすべてのモデルが含まれていることを確認します。以下のモデルがすべてサポートされています（jittorllmsはDockerのみサポートされています）：
|
|
||||||
AVAIL_LLM_MODELS = ["gpt-3.5-turbo", "api2d-gpt-3.5-turbo", "gpt-4", "api2d-gpt-4", "chatglm", "moss"] # + ["jittorllms_rwkv", "jittorllms_pangualpha", "jittorllms_llama"]
|
|
||||||
```
|
|
||||||
|
|
||||||
</p>
|
|
||||||
</details>
|
|
||||||
|
|
||||||
4. 実行する
|
|
||||||
```sh
|
|
||||||
python main.py
|
|
||||||
```
|
|
||||||
|
|
||||||
### 方法II：Dockerを使用する
|
|
||||||
|
|
||||||
0. プロジェクトのフルスケールデプロイ（これは、CUDAとLaTeXを含む大規模なイメージですが、ネットワーク速度が遅いまたはディスク容量が小さい場合はおすすめしません）
|
|
||||||
[](https://github.com/binary-husky/gpt_academic/actions/workflows/build-with-all-capacity.yml)
|
|
||||||
|
|
||||||
```sh
|
|
||||||
# docker-compose.ymlを編集し、スキーム0を残し、その他を削除してから実行する：
|
|
||||||
docker-compose up
|
|
||||||
```
|
|
||||||
|
|
||||||
1. ChatGPT + 文心一言 + sparkなどのオンラインモデルのみを含む（ほとんどの人におすすめ）
|
|
||||||
[](https://github.com/binary-husky/gpt_academic/actions/workflows/build-without-local-llms.yml)
|
|
||||||
[](https://github.com/binary-husky/gpt_academic/actions/workflows/build-with-latex.yml)
|
|
||||||
[](https://github.com/binary-husky/gpt_academic/actions/workflows/build-with-audio-assistant.yml)
|
|
||||||
|
|
||||||
```sh
|
|
||||||
# docker-compose.ymlを編集し、スキーム1を残し、その他を削除してから実行する：
|
|
||||||
docker-compose up
|
|
||||||
```
|
|
||||||
|
|
||||||
P.S. LaTeXプラグインの機能を使用する場合は、Wikiを参照してください。また、LaTeX機能を使用するためには、スキーム4またはスキーム0を直接使用することもできます。
|
|
||||||
|
|
||||||
2. ChatGPT + ChatGLM2 + MOSS + LLAMA2 + 通慧千問（Nvidia Dockerに精通している場合）
|
|
||||||
[](https://github.com/binary-husky/gpt_academic/actions/workflows/build-with-chatglm.yml)
|
|
||||||
|
|
||||||
```sh
|
|
||||||
# docker-compose.ymlを編集し、スキーム2を残し、その他を削除してから実行する：
|
|
||||||
docker-compose up
|
|
||||||
```
|
|
||||||
|
|
||||||
|
|
||||||
### 方法III：その他のデプロイメントオプション
|
|
||||||
|
|
||||||
1. **Windowsのワンクリック実行スクリプト**。
|
|
||||||
Python環境に詳しくないWindowsユーザーは、[リリース](https://github.com/binary-husky/gpt_academic/releases)からワンクリック実行スクリプトをダウンロードして、ローカルモデルのないバージョンをインストールできます。
|
|
||||||
スクリプトの貢献者は[oobabooga](https://github.com/oobabooga/one-click-installers)です。
|
|
||||||
|
|
||||||
2. 第三者のAPI、Azureなど、文心一言、星火などを使用するには、[Wikiページ](https://github.com/binary-husky/gpt_academic/wiki/Getting-Started)を参照してください。
|
|
||||||
|
|
||||||
3. クラウドサーバーでのリモートデプロイの回避策ガイドを参照してください。
|
|
||||||
[クラウドサーバーでのリモートデプロイの回避策ガイドwiki](https://github.com/binary-husky/gpt_academic/wiki/Getting-Started#%E4%BA%91%E3%82%B5%E3%83%BC%E3%83%90%E3%83%BC%E3%83%AA%E3%82%BC%E3%83%A0%E3%82%B5%E3%83%BC%E3%83%90%E3%81%AE%E3%83%AA%E3%83%A2%E3%83%BC%E3%83%88%E3%83%87%E3%83%97%E3%83%AD%E3%82%A4%E6%8C%87%E5%8D%97)
|
|
||||||
|
|
||||||
4. その他の新しいデプロイプラットフォームや方法
|
|
||||||
- Sealosを使用した[ワンクリックデプロイ](https://github.com/binary-husky/gpt_academic/issues/993)
|
|
||||||
- WSL2（Windows Subsystem for Linux）の使用方法については、[デプロイwiki-2](https://github.com/binary-husky/gpt_academic/wiki/Getting-Started)を参照してください。
|
|
||||||
- サブパス（例：`http://localhost/subpath`）でFastAPIを実行する方法については、[FastAPIの実行方法](docs/WithFastapi.md)を参照してください。
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
# 高度な使用法
|
|
||||||
### I：カスタムショートカットボタンの作成（学術的なショートカットキー）
|
|
||||||
テキストエディタで`core_functional.py`を開き、次の項目を追加し、プログラムを再起動します。（ボタンが存在する場合、プレフィックスとサフィックスはホット変更に対応しており、プログラムを再起動せずに有効にすることができます。）
|
|
||||||
例：
|
|
||||||
```
|
|
||||||
"超级英译中"： {
|
|
||||||
# プレフィックス、入力の前に追加されます。例えば、要求を記述するために使用されます。翻訳、コードの解説、校正など
|
|
||||||
"プレフィックス"： "下記の内容を中国語に翻訳し、専門用語を一つずつマークダウンテーブルで解説してください：\n\n"、
|
|
||||||
|
|
||||||
# サフィックス、入力の後に追加されます。プレフィックスと一緒に使用して、入力内容を引用符で囲むことができます。
|
|
||||||
"サフィックス"： ""、
|
|
||||||
}、
|
|
||||||
```
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/226899272-477c2134-ed71-4326-810c-29891fe4a508.png" width="500" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
### II：関数プラグインのカスタマイズ
|
|
||||||
自分の望む任意のタスクを実行するために、強力な関数プラグインを作成できます。
|
|
||||||
このプロジェクトのプラグインの作成とデバッグの難易度は非常に低く、一定のPythonの基礎知識があれば、提供されたテンプレートを参考に自分自身のプラグイン機能を実装することができます。
|
|
||||||
詳細については、[関数プラグインガイド](https://github.com/binary-husky/gpt_academic/wiki/%E5%87%BD%E6%95%B0%E6%8F%92%E4%BB%B6%E6%8C%87%E5%8D%97)を参照してください。
|
|
||||||
|
|
||||||
|
|
||||||
# 更新
|
|
||||||
### I：ダイナミック
|
|
||||||
|
|
||||||
1. 会話の保存機能。プラグインエリアで `Save Current Conversation` を呼び出すだけで、現在の会話を読み取り可能で復旧可能なhtmlファイルとして保存できます。
|
|
||||||
また、プラグインエリア（ドロップダウンメニュー）で `Load Conversation History Archive` を呼び出すことで、以前の会話を復元できます。
|
|
||||||
ヒント：ファイルを指定せずに `Load Conversation History Archive` をクリックすると、履歴のhtmlアーカイブのキャッシュを表示することができます。
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/235222390-24a9acc0-680f-49f5-bc81-2f3161f1e049.png" width="500" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
2. ⭐Latex/Arxiv論文の翻訳機能⭐
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://github.com/binary-husky/gpt_academic/assets/96192199/002a1a75-ace0-4e6a-94e2-ec1406a746f1" height="250" > ===>
|
|
||||||
<img src="https://github.com/binary-husky/gpt_academic/assets/96192199/9fdcc391-f823-464f-9322-f8719677043b" height="250" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
3. ゼロのターミナル（自然言語入力からユーザの意図を理解+他のプラグインを自動的に呼び出す）
|
|
||||||
|
|
||||||
- ステップ1：「プラグインのPDF論文の翻訳を呼び出してください、アドレスはhttps://openreview.net/pdf?id=rJl0r3R9KX」と入力します。
|
|
||||||
- ステップ2：「Zero Terminal」をクリックします。
|
|
||||||
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://github.com/binary-husky/gpt_academic/assets/96192199/66f1b044-e9ff-4eed-9126-5d4f3668f1ed" width="500" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
4. モジュール化された機能設計、シンプルなインターフェイスで強力な機能をサポートする
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/229288270-093643c1-0018-487a-81e6-1d7809b6e90f.png" height="400" >
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/227504931-19955f78-45cd-4d1c-adac-e71e50957915.png" height="400" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
5. 他のオープンソースプロジェクトの翻訳
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/226935232-6b6a73ce-8900-4aee-93f9-733c7e6fef53.png" height="250" >
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/226969067-968a27c1-1b9c-486b-8b81-ab2de8d3f88a.png" height="250" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
6. [live2d](https://github.com/fghrsh/live2d_demo)のデコレーション機能（デフォルトでは無効で、`config.py`を変更する必要があります）
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/236432361-67739153-73e8-43fe-8111-b61296edabd9.png" width="500" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
7. OpenAI画像生成
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://github.com/binary-husky/gpt_academic/assets/96192199/bc7ab234-ad90-48a0-8d62-f703d9e74665" width="500" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
8. OpenAIオーディオ解析と要約
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://github.com/binary-husky/gpt_academic/assets/96192199/709ccf95-3aee-498a-934a-e1c22d3d5d5b" width="500" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
9. Latex全体の校正と修正
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://github.com/binary-husky/gpt_academic/assets/96192199/651ccd98-02c9-4464-91e1-77a6b7d1b033" height="200" > ===>
|
|
||||||
<img src="https://github.com/binary-husky/gpt_academic/assets/96192199/476f66d9-7716-4537-b5c1-735372c25adb" height="200">
|
|
||||||
</div>
|
|
||||||
|
|
||||||
10. 言語、テーマの切り替え
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://github.com/binary-husky/gpt_academic/assets/96192199/b6799499-b6fb-4f0c-9c8e-1b441872f4e8" width="500" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
### II：バージョン:
|
|
||||||
- version 3.70（todo）: AutoGenプラグインのテーマを最適化し、一連の派生プラグインを設計する
|
|
||||||
- version 3.60: AutoGenを次世代プラグインの基盤として導入
|
|
||||||
- version 3.57: GLM3、星火v3、文心一言v4をサポート、ローカルモデルの並行バグを修正
|
|
||||||
- version 3.56: 基本機能ボタンを動的に追加、新しい報告書PDF集約ページ
|
|
||||||
- version 3.55: フロントエンドのデザインを再構築し、浮動ウィンドウとメニューバーを導入
|
|
||||||
- version 3.54: 新しい動的コードインタプリタ（Code Interpreter）の追加（未完成）
|
|
||||||
- version 3.53: 異なるテーマを動的に選択できるように、安定性の向上と複数ユーザの競合問題の解決
|
|
||||||
- version 3.50: 自然言語でこのプロジェクトのすべての関数プラグインを呼び出すことができるようになりました（ゼロのターミナル）プラグインの分類をサポートし、UIを改善し、新しいテーマを設計
|
|
||||||
- version 3.49: Baidu Qianfanプラットフォームと文心一言をサポート
|
|
||||||
- version 3.48: Alibaba DAMO Academy Tongyi Qianwen、Shanghai AI-Lab Shusheng、Xunfei Xinghuoをサポート
|
|
||||||
- version 3.46: 完全なオートモードのリアルタイム音声対話をサポート
|
|
||||||
- version 3.45: カスタムChatGLM2ファインチューニングモデルをサポート
|
|
||||||
- version 3.44: 公式にAzureをサポート、UIの使いやすさを最適化
|
|
||||||
- version 3.4: +arxiv論文の翻訳、latex論文の校閲機能
|
|
||||||
- version 3.3: +インターネット情報の総合機能
|
|
||||||
- version 3.2: 関数プラグインがさらに多くのパラメータインターフェースをサポート（会話の保存機能、任意の言語のコードの解釈、同時に任意のLLMの組み合わせを尋ねる）
|
|
||||||
- version 3.1: 複数のgptモデルに同時に質問できるようにサポートされました！ api2dをサポートし、複数のapikeyの負荷分散をサポートしました
|
|
||||||
- version 3.0: chatglmと他の小さなllmのサポート
|
|
||||||
- version 2.6: プラグインの構造を再構築し、対話性を高め、より多くのプラグインを追加しました
|
|
||||||
- version 2.5: 自己更新、ソースコード全体の要約時のテキストの長さ、トークンのオーバーフローの問題を解決しました
|
|
||||||
- version 2.4: (1)新しいPDF全文翻訳機能を追加しました。(2)入力エリアの位置を切り替えるための新しい機能を追加しました。(3)垂直レイアウトオプションを追加しました。(4)マルチスレッド関数プラグインを最適化しました。
|
|
||||||
- version 2.3: マルチスレッドの対話を強化しました
|
|
||||||
- version 2.2: 関数プラグインのホットリロードをサポート
|
|
||||||
- version 2.1: 折りたたみ式のレイアウト
|
|
||||||
- version 2.0: モジュール化された関数プラグインの導入
|
|
||||||
- version 1.0: 基本機能
|
|
||||||
|
|
||||||
GPT Academic開発者QQグループ：`610599535`
|
|
||||||
|
|
||||||
-既知の問題
|
|
||||||
- 一部のブラウザ翻訳プラグインがこのソフトウェアのフロントエンドの実行を妨げる
|
|
||||||
- 公式Gradioには互換性の問題があり、必ず`requirement.txt`を使用してGradioをインストールしてください
|
|
||||||
|
|
||||||
### III：テーマ
|
|
||||||
`THEME`オプション（`config.py`）を変更することで、テーマを変更できます
|
|
||||||
1. `Chuanhu-Small-and-Beautiful` [リンク](https://github.com/GaiZhenbiao/ChuanhuChatGPT/)
|
|
||||||
|
|
||||||
|
|
||||||
### IV：本プロジェクトの開発ブランチ
|
|
||||||
|
|
||||||
1. `master`ブランチ：メインブランチ、安定版
|
|
||||||
2. `frontier`ブランチ：開発ブランチ、テスト版
|
|
||||||
|
|
||||||
|
|
||||||
### V：参考と学習
|
|
||||||
|
|
||||||
```
|
|
||||||
コードの中には、他の優れたプロジェクトのデザインを参考にしたものが多く含まれています。順序は問いません：
|
|
||||||
|
|
||||||
# 清華ChatGLM2-6B:
|
|
||||||
https://github.com/THUDM/ChatGLM2-6B
|
|
||||||
|
|
||||||
# 清華JittorLLMs:
|
|
||||||
https://github.com/Jittor/JittorLLMs
|
|
||||||
|
|
||||||
# ChatPaper:
|
|
||||||
https://github.com/kaixindelele/ChatPaper
|
|
||||||
|
|
||||||
# Edge-GPT:
|
|
||||||
https://github.com/acheong08/EdgeGPT
|
|
||||||
|
|
||||||
# ChuanhuChatGPT:
|
|
||||||
https://github.com/GaiZhenbiao/ChuanhuChatGPT
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
# Oobaboogaワンクリックインストーラー：
|
|
||||||
https://github.com/oobabooga/one-click-installers
|
|
||||||
|
|
||||||
# その他：
|
|
||||||
https://github.com/gradio-app/gradio
|
|
||||||
https://github.com/fghrsh/live2d_demo
|
|
||||||
@@ -1,363 +0,0 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
> **참고**
|
|
||||||
>
|
|
||||||
> 이 README는 GPT 번역으로 생성되었습니다 (이 프로젝트의 플러그인에 의해 구현됨) . 100% 신뢰할 수 없으므로 번역 결과를 주의 깊게 검토하십시오.
|
|
||||||
>
|
|
||||||
> 2023.11.7: 종속성을 설치할 때, `requirements.txt`에 **지정된 버전**을 선택하십시오. 설치 명령어: `pip install -r requirements.txt`.
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
# <div align=center><img src="logo.png" width="40"> GPT 학술 최적화 (GPT Academic)</div>
|
|
||||||
|
|
||||||
**이 프로젝트가 마음에 드신다면, Star를 부탁드립니다. 편리한 단축키나 플러그인을 발견하셨다면 Pull Request를 환영합니다!**
|
|
||||||
GPT를 사용하여 이 프로젝트를 임의의 언어로 번역하려면 [`multi_language.py`](multi_language.py)를 읽고 실행하십시오 (실험적).
|
|
||||||
|
|
||||||
|
|
||||||
> **참고**
|
|
||||||
>
|
|
||||||
> 1. **강조 표시**된 플러그인 (버튼)만 파일을 읽을 수 있습니다. 일부 플러그인은 플러그인 영역의 **드롭다운 메뉴**에 있습니다. 또한 새로운 플러그인에 대한 모든 PR을 환영하며, 이를 **가장 우선적**으로 처리합니다.
|
|
||||||
>
|
|
||||||
> 2. 이 프로젝트의 각 파일의 기능은 [자체 분석 보고서 `self_analysis.md`](https://github.com/binary-husky/gpt_academic/wiki/GPT‐Academic%EC%A0%9C%ED%94%84%EB%AA%85%EC%84%B1%EB%B0%A9%EC%8B%9D%EC%9D%98_%EA%B2%B0%EA%B3%BC)에서 자세히 설명되어 있습니다. 버전이 반복됨에 따라, 관련 기능 플러그인을 언제든지 클릭하여 GPT를 호출하여 프로젝트의 자체 분석 보고서를 다시 생성할 수 있습니다. 자주 묻는 질문은 [`위키`](https://github.com/binary-husky/gpt_academic/wiki)를 참조하십시오. [일반적인 설치 방법](#installation) | [원클릭 설치 스크립트](https://github.com/binary-husky/gpt_academic/releases) | [설정 설명서](https://github.com/binary-husky/gpt_academic/wiki/%EC%84%A4%EC%A0%95%EC%82%AC%EB%AA%85_%EA%B0%84%EB%8B%A8_%EC%84%B8%ED%8A%B8%EB%B2%84_%EC%B6%94%EA%B0%80)
|
|
||||||
|
|
||||||
|
|
||||||
> 3. 이 프로젝트는 ChatGLM 등 대형 언어 모델 (ChatGLM 등) 실행을 지원하고 권장합니다. 여러 개의 API 키를 동시에 사용할 수 있으며, 구성 파일에 `API_KEY="openai-key1,openai-key2,azure-key3,api2d-key4"`와 같이 입력할 수 있습니다. `API_KEY`를 일시적으로 변경해야 하는 경우, 입력 영역에 임시 `API_KEY`를 입력한 다음 Enter 키를 누르면 적용됩니다.
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
<div align="center">
|
|
||||||
|
|
||||||
기능 (⭐= 최근 추가 기능) | 설명
|
|
||||||
--- | ---
|
|
||||||
⭐[새 모델 추가](https://github.com/binary-husky/gpt_academic/wiki/%E5%A6%82%E4%BD%95%E5%88%87%E6%8D%A2%E6%A8%A1%E5%9E%8B)! | Baidu [Qianfan](https://cloud.baidu.com/doc/WENXINWORKSHOP/s/Nlks5zkzu)와 Wenxin Yiyan, [Tongyi Qianwen](https://modelscope.cn/models/qwen/Qwen-7B-Chat/summary), Shanghai AI-Lab [Shusheng](https://github.com/InternLM/InternLM), Xunfei [Star](https://xinghuo.xfyun.cn/), [LLaMa2](https://huggingface.co/meta-llama/Llama-2-7b-chat-hf), Zhipu API, DALLE3
|
|
||||||
문체 개선, 번역, 코드 설명 | 일괄적인 문체 개선, 번역, 논문 문법 오류 탐색, 코드 설명
|
|
||||||
[사용자 정의 단축키](https://www.bilibili.com/video/BV14s4y1E7jN) | 사용자 정의 단축키 지원
|
|
||||||
모듈화 설계 | 사용자 정의 가능한 강력한 [플러그인](https://github.com/binary-husky/gpt_academic/tree/master/crazy_functions) 지원, 플러그인 지원 [핫 업데이트](https://github.com/binary-husky/gpt_academic/wiki/%E5%87%BD%E6%95%B0%E6%8F%92%E4%BB%B6%E6%8C%87%E5%8D%97)
|
|
||||||
[프로그램 분석](https://www.bilibili.com/video/BV1cj411A7VW) | [플러그인] 한 번에 Python/C/C++/Java/Lua/... 프로젝트 트리를 분석하거나 [자체 분석](https://www.bilibili.com/video/BV1cj411A7VW)
|
|
||||||
논문 읽기, 논문 [번역](https://www.bilibili.com/video/BV1KT411x7Wn) | [플러그인] LaTeX/PDF 논문 전문을 읽고 요약 생성
|
|
||||||
LaTeX 전체 [번역](https://www.bilibili.com/video/BV1nk4y1Y7Js/), [개선](https://www.bilibili.com/video/BV1FT411H7c5/) | [플러그인] LaTeX 논문 번역 또는 개선
|
|
||||||
일괄 주석 생성 | [플러그인] 함수 주석 일괄 생성
|
|
||||||
Markdown [한 / 영 번역](https://www.bilibili.com/video/BV1yo4y157jV/) | 위의 5개 언어로 작성된 [README](https://github.com/binary-husky/gpt_academic/blob/master/docs/README_EN.md)를 살펴보셨나요?
|
|
||||||
채팅 분석 보고서 생성 | [플러그인] 실행 후 요약 보고서 자동 생성
|
|
||||||
[PDF 논문 전체 번역](https://www.bilibili.com/video/BV1KT411x7Wn) 기능 | [플러그인] PDF 논문 제목 및 요약 추출 + 전체 번역 (멀티 스레드)
|
|
||||||
[Arxiv 도우미](https://www.bilibili.com/video/BV1LM4y1279X) | [플러그인] arxiv 논문 url 입력시 요약 번역 + PDF 다운로드
|
|
||||||
LaTeX 논문 일괄 교정 | [플러그인] Grammarly를 모사하여 LaTeX 논문에 대한 문법 및 맞춤법 오류 교정 + 대조 PDF 출력
|
|
||||||
[Google 학술 통합 도우미](https://www.bilibili.com/video/BV19L411U7ia) | 임의의 Google 학술 검색 페이지 URL을 지정하여 gpt가 [related works를 작성](https://www.bilibili.com/video/BV1GP411U7Az/)하게 해주세요.
|
|
||||||
인터넷 정보 집계 + GPT | [플러그인] [인터넷에서 정보를 가져와서](https://www.bilibili.com/video/BV1om4y127ck) 질문에 대답하도록 GPT를 자동화하세요. 정보가 절대로 오래되지 않도록 해줍니다.
|
|
||||||
⭐Arxiv 논문 세심한 번역 ([Docker](https://github.com/binary-husky/gpt_academic/pkgs/container/gpt_academic_with_latex)) | [플러그인] [arxiv 논문을 고품질 번역으로](https://www.bilibili.com/video/BV1dz4y1v77A/) 번역하는 최고의 도구
|
|
||||||
⭐[실시간 음성 대화 입력](https://github.com/binary-husky/gpt_academic/blob/master/docs/use_audio.md) | [플러그인] 비동기적으로 [오디오를 모니터링](https://www.bilibili.com/video/BV1AV4y187Uy/)하여 문장을 자동으로 분절하고 대답 시기를 자동으로 찾습니다.
|
|
||||||
수식/이미지/표 표시 | [tex 형식 및 렌더링 형식](https://user-images.githubusercontent.com/96192199/230598842-1d7fcddd-815d-40ee-af60-baf488a199df.png)의 수식을 동시에 표시하며, 수식 및 코드 하이라이트 지원
|
|
||||||
⭐AutoGen multi-agent 플러그인 | [플러그인] Microsoft AutoGen을 활용하여 여러 개의 에이전트가 지능적으로 발생하는 가능성을 탐색하세요!
|
|
||||||
다크 모드 주제 지원 | 브라우저의 URL 뒤에 ```/?__theme=dark```를 추가하여 다크 모드로 전환하세요.
|
|
||||||
[다양한 LLM 모델](https://www.bilibili.com/video/BV1wT411p7yf) 지원 | GPT3.5, GPT4, [Tsinghua ChatGLM2](https://github.com/THUDM/ChatGLM2-6B), [Fudan MOSS](https://github.com/OpenLMLab/MOSS)을 함께 사용하는 느낌은 좋을 것입니다, 그렇지 않습니까?
|
|
||||||
⭐ChatGLM2 fine-tuned 모델 | ChatGLM2 fine-tuned 모델 로드를 지원하며, ChatGLM2 fine-tuned 보조 플러그인 제공
|
|
||||||
더 많은 LLM 모델 연결, [huggingface 배포](https://huggingface.co/spaces/qingxu98/gpt-academic) 지원 | Newbing 인터페이스(신 밍), Tsinghua [Jittorllms](https://github.com/Jittor/JittorLLMs) 도입, [LLaMA](https://github.com/facebookresearch/llama)와 [Pangu-alpha](https://openi.org.cn/pangu/)를 지원합니다.
|
|
||||||
⭐[void-terminal](https://github.com/binary-husky/void-terminal) 패키지 | GUI에서 독립, Python에서 이 프로젝트의 모든 함수 플러그인을 직접 호출 (개발 중)
|
|
||||||
⭐Void 터미널 플러그인 | [플러그인] 자연어로 이 프로젝트의 다른 플러그인을 직접 영속합니다.
|
|
||||||
기타 새로운 기능 소개 (이미지 생성 등) …… | 본 문서 맨 끝 참조 ……
|
|
||||||
</div>
|
|
||||||
|
|
||||||
|
|
||||||
- 새로운 인터페이스(`config.py`의 LAYOUT 옵션 수정으로 "왼쪽-오른쪽 레이아웃"과 "위-아래 레이아웃"을 전환할 수 있음)
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://github.com/binary-husky/gpt_academic/assets/96192199/d81137c3-affd-4cd1-bb5e-b15610389762" width="700" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
|
|
||||||
- 모든 버튼은 functional.py를 동적으로 읽어 생성되므로 원하는대로 사용자 정의 기능을 추가할 수 있으며 클립 보드를 해제할 수 있습니다.
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/231975334-b4788e91-4887-412f-8b43-2b9c5f41d248.gif" width="700" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
- 문체 개선/오류 수정
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/231980294-f374bdcb-3309-4560-b424-38ef39f04ebd.gif" width="700" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
- If the output contains equations, they will be displayed in both tex format and rendered format for easy copying and reading.
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/230598842-1d7fcddd-815d-40ee-af60-baf488a199df.png" width="700" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
- Don't feel like looking at the project code? Just give it to ChatGPT and let it dazzle you.
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/226935232-6b6a73ce-8900-4aee-93f9-733c7e6fef53.png" width="700" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
- Mix and match multiple powerful language models (ChatGLM + OpenAI-GPT3.5 + [API2D](https://api2d.com/)-GPT4)
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/232537274-deca0563-7aa6-4b5d-94a2-b7c453c47794.png" width="700" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
# Installation
|
|
||||||
### Installation Method I: Run Directly (Windows, Linux or MacOS)
|
|
||||||
|
|
||||||
1. Download the project
|
|
||||||
```sh
|
|
||||||
git clone --depth=1 https://github.com/binary-husky/gpt_academic.git
|
|
||||||
cd gpt_academic
|
|
||||||
```
|
|
||||||
|
|
||||||
2. Configure API_KEY
|
|
||||||
|
|
||||||
In `config.py`, configure the API KEY and other settings, [click here to view special network environment configuration methods](https://github.com/binary-husky/gpt_academic/issues/1). [Wiki page](https://github.com/binary-husky/gpt_academic/wiki/项目配置说明)。
|
|
||||||
|
|
||||||
" The program will first check if there is a confidential configuration file named `config_private.py` and use its configuration to override the configuration with the same name in `config.py`. If you can understand this reading logic, we strongly recommend that you create a new configuration file named `config_private.py` next to `config.py` and move (copy) the configuration from `config.py` to `config_private.py` (only copy the modified configuration items). "
|
|
||||||
|
|
||||||
" You can configure the project through `environment variables`. The format of the environment variables can be found in the `docker-compose.yml` file or our [Wiki page](https://github.com/binary-husky/gpt_academic/wiki/项目配置说明). The priority of the configuration reading is: `environment variables` > `config_private.py` > `config.py`. "
|
|
||||||
|
|
||||||
3. Install dependencies
|
|
||||||
```sh
|
|
||||||
# (Option I: if familiar with python, python>=3.9) Note: Use the official pip source or Aliyun pip source. Temporary switching source method: python -m pip install -r requirements.txt -i https://mirrors.aliyun.com/pypi/simple/
|
|
||||||
python -m pip install -r requirements.txt
|
|
||||||
|
|
||||||
# (Option II: using Anaconda) The steps are similar (https://www.bilibili.com/video/BV1rc411W7Dr):
|
|
||||||
conda create -n gptac_venv python=3.11 # Create an Anaconda environment
|
|
||||||
conda activate gptac_venv # Activate the Anaconda environment
|
|
||||||
python -m pip install -r requirements.txt # This step is the same as the pip installation step
|
|
||||||
```
|
|
||||||
|
|
||||||
|
|
||||||
<details><summary>Click here to expand if you need support for Tsinghua ChatGLM2/Fudan MOSS/RWKV backend</summary>
|
|
||||||
<p>
|
|
||||||
|
|
||||||
[Optional Step] If you need support for Tsinghua ChatGLM2/Fudan MOSS as the backend, you need to install additional dependencies (Prerequisites: Familiar with Python + Have used Pytorch + Sufficient computer configuration):
|
|
||||||
```sh
|
|
||||||
# [Optional Step I] Support for Tsinghua ChatGLM2. Note for Tsinghua ChatGLM: If you encounter the error "Call ChatGLM fail cannot load ChatGLM parameters", refer to the following: 1: The default installation above is torch+cpu version. To use cuda, uninstall torch and reinstall torch+cuda; 2: If you cannot load the model due to insufficient computer configuration, you can modify the model precision in request_llm/bridge_chatglm.py, change AutoTokenizer.from_pretrained("THUDM/chatglm-6b", trust_remote_code=True) to AutoTokenizer.from_pretrained("THUDM/chatglm-6b-int4", trust_remote_code=True)
|
|
||||||
python -m pip install -r request_llms/requirements_chatglm.txt
|
|
||||||
|
|
||||||
# [Optional Step II] Support for Fudan MOSS
|
|
||||||
python -m pip install -r request_llms/requirements_moss.txt
|
|
||||||
git clone --depth=1 https://github.com/OpenLMLab/MOSS.git request_llms/moss # When executing this line of code, make sure you are in the project root path
|
|
||||||
|
|
||||||
# [Optional Step III] Support for RWKV Runner
|
|
||||||
Refer to the wiki: https://github.com/binary-husky/gpt_academic/wiki/%E9%80%82%E9%85%8DRWKV-Runner
|
|
||||||
|
|
||||||
# [Optional Step IV] Make sure that the AVAIL_LLM_MODELS in the config.py configuration file includes the expected models. The currently supported models are as follows (the jittorllms series only supports the docker solution):
|
|
||||||
AVAIL_LLM_MODELS = ["gpt-3.5-turbo", "api2d-gpt-3.5-turbo", "gpt-4", "api2d-gpt-4", "chatglm", "moss"] # + ["jittorllms_rwkv", "jittorllms_pangualpha", "jittorllms_llama"]
|
|
||||||
```
|
|
||||||
|
|
||||||
</p>
|
|
||||||
</details>
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
4. Run
|
|
||||||
```sh
|
|
||||||
python main.py
|
|
||||||
```
|
|
||||||
|
|
||||||
### Installation Method II: Use Docker
|
|
||||||
|
|
||||||
0. Deploy all the capabilities of the project (this is a large image that includes cuda and latex. However, it is not recommended if your internet speed is slow or your hard disk is small)
|
|
||||||
[](https://github.com/binary-husky/gpt_academic/actions/workflows/build-with-all-capacity.yml)
|
|
||||||
|
|
||||||
``` sh
|
|
||||||
# Modify docker-compose.yml, keep scheme 0 and delete the others. Then run:
|
|
||||||
docker-compose up
|
|
||||||
```
|
|
||||||
|
|
||||||
1. ChatGPT+Random Quotes+Wikipedia Summary+Spark and other online models (recommended for most people)
|
|
||||||
[](https://github.com/binary-husky/gpt_academic/actions/workflows/build-without-local-llms.yml)
|
|
||||||
[](https://github.com/binary-husky/gpt_academic/actions/workflows/build-with-latex.yml)
|
|
||||||
[](https://github.com/binary-husky/gpt_academic/actions/workflows/build-with-audio-assistant.yml)
|
|
||||||
|
|
||||||
``` sh
|
|
||||||
# Modify docker-compose.yml, keep scheme 1 and delete the others. Then run:
|
|
||||||
docker-compose up
|
|
||||||
```
|
|
||||||
|
|
||||||
P.S. If you need the Latex plugin feature, please refer to the Wiki. Additionally, you can also use scheme 4 or scheme 0 directly to get the Latex feature.
|
|
||||||
|
|
||||||
2. ChatGPT + ChatGLM2 + MOSS + LLAMA2 + Thousand Questions (Requires familiarity with [Nvidia Docker](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html#installing-on-ubuntu-and-debian) runtime)
|
|
||||||
[](https://github.com/binary-husky/gpt_academic/actions/workflows/build-with-chatglm.yml)
|
|
||||||
|
|
||||||
``` sh
|
|
||||||
# Modify docker-compose.yml, keep scheme 2 and delete the others. Then run:
|
|
||||||
docker-compose up
|
|
||||||
```
|
|
||||||
|
|
||||||
|
|
||||||
### Installation Method III: Other Deployment Methods
|
|
||||||
1. **One-click run script for Windows**.
|
|
||||||
Windows users who are completely unfamiliar with the Python environment can download the one-click run script without local models from the [Release](https://github.com/binary-husky/gpt_academic/releases) section.
|
|
||||||
The script contribution comes from [oobabooga](https://github.com/oobabooga/one-click-installers).
|
|
||||||
|
|
||||||
2. Use third-party APIs, Azure, etc., Random Quotes, Spark, etc., see the [Wiki page](https://github.com/binary-husky/gpt_academic/wiki/项目配置说明).
|
|
||||||
|
|
||||||
3. Pitfall guide for remote deployment on cloud servers.
|
|
||||||
Please visit the [cloud server remote deployment wiki](https://github.com/binary-husky/gpt_academic/wiki/%E4%BA%91%E6%9C%8D%E5%8A%A1%E5%99%A8%E8%BF%9C%E7%A8%8B%E9%83%A8%E7%BD%B2%E6%8C%87%E5%8D%97)
|
|
||||||
|
|
||||||
4. Some new deployment platforms or methods
|
|
||||||
- Use Sealos for [one-click deployment](https://github.com/binary-husky/gpt_academic/issues/993).
|
|
||||||
- Use WSL2 (Windows Subsystem for Linux). Please visit [deployment wiki-2](https://github.com/binary-husky/gpt_academic/wiki/%E4%BD%BF%E7%94%A8WSL2%EF%BC%88Windows-Subsystem-for-Linux-%E5%AD%90%E7%B3%BB%E7%BB%9F%EF%BC%89%E9%83%A8%E7%BD%B2)
|
|
||||||
- How to run in a subpath (such as `http://localhost/subpath`). Please refer to [FastAPI running instructions](docs/WithFastapi.md)
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
# 고급 사용법
|
|
||||||
### I: 사용자 정의 바로 가기 버튼 추가 (학술 단축키)
|
|
||||||
임의의 텍스트 편집기로 `core_functional.py` 파일을 열고 다음과 같은 항목을 추가한 다음 프로그램을 다시 시작하십시오. (이미 버튼이 있는 경우에는 접두사와 접미사를 실시간으로 수정할 수 있으므로 프로그램을 다시 시작할 필요가 없습니다.)
|
|
||||||
예시:
|
|
||||||
```
|
|
||||||
"초급영문 번역": {
|
|
||||||
# 접두사, 입력 내용 앞에 추가됩니다. 예를 들어 요구 사항을 설명하는 데 사용됩니다. 예를 들어 번역, 코드 설명, 교정 등
|
|
||||||
"Prefix": "다음 내용을 한국어로 번역하고 전문 용어에 대한 설명을 적용한 마크다운 표를 사용하세요:\n\n",
|
|
||||||
|
|
||||||
# 접미사, 입력 내용 뒤에 추가됩니다. 예를 들어 접두사와 함께 입력 내용을 따옴표로 감쌀 수 있습니다.
|
|
||||||
"Suffix": "",
|
|
||||||
},
|
|
||||||
```
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/226899272-477c2134-ed71-4326-810c-29891fe4a508.png" width="500" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
### II: 사용자 정의 함수 플러그인
|
|
||||||
원하는 작업을 수행하기 위해 능력있는 함수 플러그인을 작성하세요.
|
|
||||||
이 프로젝트의 플러그인 작성 및 디버깅은 난이도가 낮으며, 일정한 Python 기본 지식만 있으면 우리가 제공하는 템플릿을 본따서 고유한 플러그인 기능을 구현할 수 있습니다.
|
|
||||||
자세한 내용은 [함수 플러그인 가이드](https://github.com/binary-husky/gpt_academic/wiki/%E5%87%BD%E6%95%B0%E6%8F%92%E4%BB%B6%E6%8C%87%E5%8D%97)를 참조하세요.
|
|
||||||
|
|
||||||
|
|
||||||
# 업데이트
|
|
||||||
### I: 다이나믹
|
|
||||||
|
|
||||||
1. 대화 저장 기능. 플러그인 영역에서 '현재 대화 저장'을 호출하여 현재 대화를 볼 수 있고, html 파일을 복구할 수 있습니다.
|
|
||||||
또한 플러그인 영역에서 '대화 기록 불러오기'를 호출하여 이전 대화를 복원할 수 있습니다.
|
|
||||||
팁: 파일을 지정하지 않고 '대화 기록 불러오기'를 바로 클릭하면 이전 html 기록 캐시를 볼 수 있습니다.
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/235222390-24a9acc0-680f-49f5-bc81-2f3161f1e049.png" width="500" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
2. ⭐Latex/Arxiv 논문 번역 기능⭐
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://github.com/binary-husky/gpt_academic/assets/96192199/002a1a75-ace0-4e6a-94e2-ec1406a746f1" height="250" > ===>
|
|
||||||
<img src="https://github.com/binary-husky/gpt_academic/assets/96192199/9fdcc391-f823-464f-9322-f8719677043b" height="250" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
3. 빈 터미널 (자연어 입력에서 사용자 의도 이해 + 자동 플러그인 호출)
|
|
||||||
|
|
||||||
- 단계 1: "플러그인을 사용하여 PDF 논문을 번역하십시오. 주소는 https://openreview.net/pdf?id=rJl0r3R9KX입니다." 입력
|
|
||||||
- 단계 2: "빈 터미널" 클릭
|
|
||||||
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://github.com/binary-husky/gpt_academic/assets/96192199/66f1b044-e9ff-4eed-9126-5d4f3668f1ed" width="500" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
4. 모듈화된 기능 디자인, 간단한 인터페이스로 강력한 기능 제공
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/229288270-093643c1-0018-487a-81e6-1d7809b6e90f.png" height="400" >
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/227504931-19955f78-45cd-4d1c-adac-e71e50957915.png" height="400" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
5. 다른 오픈 소스 프로젝트 번역
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/226935232-6b6a73ce-8900-4aee-93f9-733c7e6fef53.png" height="250" >
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/226969067-968a27c1-1b9c-486b-8b81-ab2de8d3f88a.png" height="250" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
6. [live2d](https://github.com/fghrsh/live2d_demo)의 작은 기능 추가 (기본 설정은 닫혀 있으며, `config.py`를 수정해야 합니다.)
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/236432361-67739153-73e8-43fe-8111-b61296edabd9.png" width="500" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
7. OpenAI 이미지 생성
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://github.com/binary-husky/gpt_academic/assets/96192199/bc7ab234-ad90-48a0-8d62-f703d9e74665" width="500" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
8. OpenAI 오디오 분석 및 요약
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://github.com/binary-husky/gpt_academic/assets/96192199/709ccf95-3aee-498a-934a-e1c22d3d5d5b" width="500" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
9. Latex 전체 교정 오류
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://github.com/binary-husky/gpt_academic/assets/96192199/651ccd98-02c9-4464-91e1-77a6b7d1b033" height="200" > ===>
|
|
||||||
<img src="https://github.com/binary-husky/gpt_academic/assets/96192199/476f66d9-7716-4537-b5c1-735372c25adb" height="200">
|
|
||||||
</div>
|
|
||||||
|
|
||||||
10. 언어, 테마 변경
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://github.com/binary-husky/gpt_academic/assets/96192199/b6799499-b6fb-4f0c-9c8e-1b441872f4e8" width="500" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
### II: 버전:
|
|
||||||
- 버전 3.70 (예정): AutoGen 플러그인 테마 개선 및 다른 테마 플러그인 디자인
|
|
||||||
- 버전 3.60: AutoGen을 새로운 세대 플러그인의 기반으로 도입
|
|
||||||
- 버전 3.57: GLM3, Starfire v3, 文心一言 v4 지원, 로컬 모델의 동시성 버그 수정
|
|
||||||
- 버전 3.56: 동적으로 기본 기능 버튼 추가, 새로운 보고서 PDF 요약 페이지
|
|
||||||
- 버전 3.55: 프론트 엔드 인터페이스 리팩토링, 화면 따라다니는 윈도우 및 메뉴 바 도입
|
|
||||||
- 버전 3.54: 새로운 동적 코드 해석기 (Code Interpreter) 추가 (완벽하게 완성되지 않음)
|
|
||||||
- 버전 3.53: 다른 인터페이스 테마 동적 선택 기능 추가, 안정성 향상 및 다중 사용자 충돌 문제 해결
|
|
||||||
- 버전 3.50: 자연어로 이 프로젝트의 모든 함수 플러그인을 호출하는 기능 (빈 터미널) 추가, 플러그인 분류 지원, UI 개선, 새로운 테마 설계
|
|
||||||
- 버전 3.49: Baidu Qianfan 플랫폼 및 문심일언 지원
|
|
||||||
- 버전 3.48: Ali DameiYuan Sematic Query, Shanghai AI-Lab Shusheng, Xunfei Starfire 지원
|
|
||||||
- 버전 3.46: 완전 자동 운전 가능한 실시간 음성 대화 지원
|
|
||||||
- 버전 3.45: 사용자 정의 ChatGLM2 fine-tuning 모델 지원
|
|
||||||
- 버전 3.44: Azure 정식 지원, 인터페이스의 사용 편의성 개선
|
|
||||||
- 버전 3.4: +arxiv 논문 번역, latex 논문 교정 기능 추가
|
|
||||||
- 버전 3.3: +인터넷 정보 종합 기능
|
|
||||||
- 버전 3.2: 함수 플러그인이 더 많은 매개변수 인터페이스를 지원합니다 (대화 저장 기능, 임의의 언어 코드 해석 + 임의의 LLM 조합을 동시에 요청)
|
|
||||||
- 버전 3.1: 여러 GPT 모델에 동시에 질문할 수 있는 기능 추가! api2d 지원, 여러 개의 apikey 부하 균형 조정 지원
|
|
||||||
- 버전 3.0: chatglm 및 기타 소규모 llm 지원
|
|
||||||
- 버전 2.6: 플러그인 구조를 재구성하여 상호 작용성 향상, 더 많은 플러그인 추가
|
|
||||||
- 버전 2.5: 자동 업데이트, 소스 코드 요약 중 텍스트가 너무 길고 토큰이 오버플로되는 문제 해결
|
|
||||||
- 버전 2.4: (1)PDF 전체 번역 기능 추가; (2)입력 영역 위치 전환 기능 추가; (3)수직 레이아웃 옵션 추가; (4)멀티 스레드 함수 플러그인 최적화
|
|
||||||
- 버전 2.3: 멀티 스레드 상호 작용성 강화
|
|
||||||
- 버전 2.2: 함수 플러그인의 핫 리로드 지원
|
|
||||||
- 버전 2.1: 접을 수 있는 레이아웃
|
|
||||||
- 버전 2.0: 모듈화 함수 플러그인 도입
|
|
||||||
- 버전 1.0: 기본 기능
|
|
||||||
|
|
||||||
GPT Academic 개발자 QQ 그룹: `610599535`
|
|
||||||
- 알려진 문제
|
|
||||||
- 특정 웹 브라우저 번역 플러그인이 이 소프트웨어의 프론트엔드 실행에 방해가 되는 경우가 있습니다.
|
|
||||||
- 공식 Gradio에는 호환성 문제가 많기 때문에 `requirement.txt`를 사용하여 Gradio를 설치하십시오.
|
|
||||||
|
|
||||||
### III: 테마
|
|
||||||
`THEME` 옵션 (`config.py`)을 수정하여 테마를 변경할 수 있습니다.
|
|
||||||
1. `Chuanhu-Small-and-Beautiful` [URL](https://github.com/GaiZhenbiao/ChuanhuChatGPT/)
|
|
||||||
|
|
||||||
|
|
||||||
### IV: 이 프로젝트의 개발 브랜치
|
|
||||||
|
|
||||||
1. `master` 브랜치: 메인 브랜치, 안정 버전
|
|
||||||
2. `frontier` 브랜치: 개발 브랜치, 테스트 버전
|
|
||||||
|
|
||||||
|
|
||||||
### V: 참고 및 학습
|
|
||||||
|
|
||||||
```
|
|
||||||
코드에서는 다른 우수한 프로젝트의 디자인을 많이 참고했습니다. 순서는 문제 없이 나열됩니다:
|
|
||||||
|
|
||||||
# 清华ChatGLM2-6B:
|
|
||||||
https://github.com/THUDM/ChatGLM2-6B
|
|
||||||
|
|
||||||
# 清华JittorLLMs:
|
|
||||||
https://github.com/Jittor/JittorLLMs
|
|
||||||
|
|
||||||
# ChatPaper:
|
|
||||||
https://github.com/kaixindelele/ChatPaper
|
|
||||||
|
|
||||||
# Edge-GPT:
|
|
||||||
https://github.com/acheong08/EdgeGPT
|
|
||||||
|
|
||||||
# ChuanhuChatGPT:
|
|
||||||
https://github.com/GaiZhenbiao/ChuanhuChatGPT
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
# Oobabooga 원 클릭 설치 프로그램:
|
|
||||||
https://github.com/oobabooga/one-click-installers
|
|
||||||
|
|
||||||
# 더보기：
|
|
||||||
https://github.com/gradio-app/gradio
|
|
||||||
https://github.com/fghrsh/live2d_demo
|
|
||||||
@@ -1,357 +0,0 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
> **Nota**
|
|
||||||
>
|
|
||||||
> Este README foi traduzido pelo GPT (implementado por um plugin deste projeto) e não é 100% confiável. Por favor, verifique cuidadosamente o resultado da tradução.
|
|
||||||
>
|
|
||||||
> 7 de novembro de 2023: Ao instalar as dependências, favor selecionar as **versões especificadas** no `requirements.txt`. Comando de instalação: `pip install -r requirements.txt`.
|
|
||||||
|
|
||||||
# <div align=center><img src="logo.png" width="40"> GPT Acadêmico</div>
|
|
||||||
|
|
||||||
**Se você gosta deste projeto, por favor, dê uma estrela nele. Se você inventou atalhos de teclado ou plugins úteis, fique à vontade para criar pull requests!**
|
|
||||||
Para traduzir este projeto para qualquer idioma utilizando o GPT, leia e execute [`multi_language.py`](multi_language.py) (experimental).
|
|
||||||
|
|
||||||
> **Nota**
|
|
||||||
>
|
|
||||||
> 1. Observe que apenas os plugins (botões) marcados em **destaque** são capazes de ler arquivos, alguns plugins estão localizados no **menu suspenso** do plugin area. Também damos boas-vindas e prioridade máxima a qualquer novo plugin via PR.
|
|
||||||
>
|
|
||||||
> 2. As funcionalidades de cada arquivo deste projeto estão detalhadamente explicadas em [autoanálise `self_analysis.md`](https://github.com/binary-husky/gpt_academic/wiki/GPT‐Academic项目自译解报告). Com a iteração das versões, você também pode clicar nos plugins de funções relevantes a qualquer momento para chamar o GPT para regerar o relatório de autonálise do projeto. Perguntas frequentes [`wiki`](https://github.com/binary-husky/gpt_academic/wiki) | [Método de instalação convencional](#installation) | [Script de instalação em um clique](https://github.com/binary-husky/gpt_academic/releases) | [Explicação de configuração](https://github.com/binary-husky/gpt_academic/wiki/项目配置说明)。
|
|
||||||
>
|
|
||||||
> 3. Este projeto é compatível e encoraja o uso de modelos de linguagem chineses, como ChatGLM. Vários api-keys podem ser usados simultaneamente, podendo ser especificados no arquivo de configuração como `API_KEY="openai-key1,openai-key2,azure-key3,api2d-key4"`. Quando precisar alterar temporariamente o `API_KEY`, insira o `API_KEY` temporário na área de entrada e pressione Enter para que ele seja efetivo.
|
|
||||||
|
|
||||||
|
|
||||||
<div align="center">
|
|
||||||
|
|
||||||
Funcionalidades (⭐= funcionalidade recentemente adicionada) | Descrição
|
|
||||||
--- | ---
|
|
||||||
⭐[Integração com novos modelos](https://github.com/binary-husky/gpt_academic/wiki/%E5%A6%82%E4%BD%95%E5%88%87%E6%8D%A2%E6%A8%A1%E5%9E%8B)！ | [Qianfan](https://cloud.baidu.com/doc/WENXINWORKSHOP/s/Nlks5zkzu) da Baidu, Wenxin e [Tongyi Qianwen](https://modelscope.cn/models/qwen/Qwen-7B-Chat/summary), [Shusheng](https://github.com/InternLM/InternLM) da Shanghai AI-Lab, [Xinghuo](https://xinghuo.xfyun.cn/) da Iflytek, [LLaMa2](https://huggingface.co/meta-llama/Llama-2-7b-chat-hf), Zhipu API, DALLE3
|
|
||||||
Aprimoramento, tradução, explicação de códigos | Aprimoramento com um clique, tradução, busca de erros gramaticais em artigos e explicação de códigos
|
|
||||||
[Atalhos de teclado personalizados](https://www.bilibili.com/video/BV14s4y1E7jN) | Suporte para atalhos de teclado personalizados
|
|
||||||
Design modular | Suporte a plugins poderosos e personalizáveis, plugins com suporte a [atualização a quente](https://github.com/binary-husky/gpt_academic/wiki/%E5%87%BD%E6%95%B0%E6%8F%92%E4%BB%B6%E6%8C%87%E5%8D%97)
|
|
||||||
[Análise de código](https://www.bilibili.com/video/BV1cj411A7VW) | [Plugin] Análise instantânea da estrutura de projetos em Python/C/C++/Java/Lua/... ou [autoanálise](https://www.bilibili.com/video/BV1cj411A7VW)
|
|
||||||
Leitura de artigos, [tradução](https://www.bilibili.com/video/BV1KT411x7Wn) de artigos | [Plugin] Interpretação instantânea de artigos completos em latex/pdf e geração de resumos
|
|
||||||
Tradução completa de artigos em latex [PDF](https://www.bilibili.com/video/BV1nk4y1Y7Js/), [aprimoramento](https://www.bilibili.com/video/BV1FT411H7c5/) | [Plugin] Tradução completa ou aprimoramento de artigos em latex com um clique
|
|
||||||
Geração em lote de comentários | [Plugin] Geração em lote de comentários de funções com um clique
|
|
||||||
Tradução (inglês-chinês) de Markdown | [Plugin] Você já viu o [README](https://github.com/binary-husky/gpt_academic/blob/master/docs/README_EN.md) nas 5 línguas acima?
|
|
||||||
Criação de relatório de análise de bate-papo | [Plugin] Geração automática de relatório de resumo após a execução
|
|
||||||
Tradução [completa de artigos em PDF](https://www.bilibili.com/video/BV1KT411x7Wn) | [Plugin] Extração de título e resumo de artigos em PDF + tradução completa (multithreading)
|
|
||||||
Auxiliar Arxiv | [Plugin] Insira o URL de um artigo Arxiv para traduzir o resumo + baixar o PDF com um clique
|
|
||||||
Correção automática de artigos em latex | [Plugin] Correções gramaticais e ortográficas de artigos em latex semelhante ao Grammarly + saída PDF comparativo
|
|
||||||
Auxiliar Google Scholar | [Plugin] Insira qualquer URL da busca do Google Acadêmico e deixe o GPT [escrever trabalhos relacionados](https://www.bilibili.com/video/BV1GP411U7Az/) para você
|
|
||||||
Agregação de informações da Internet + GPT | [Plugin] Capturar informações da Internet e obter respostas de perguntas com o GPT em um clique, para que as informações nunca fiquem desatualizadas
|
|
||||||
⭐Tradução refinada de artigos do Arxiv ([Docker](https://github.com/binary-husky/gpt_academic/pkgs/container/gpt_academic_with_latex)) | [Plugin] Tradução de alta qualidade de artigos do Arxiv com um clique, a melhor ferramenta de tradução de artigos atualmente
|
|
||||||
⭐Entrada de conversa de voz em tempo real | [Plugin] Monitoramento de áudio [assíncrono](https://www.bilibili.com/video/BV1AV4y187Uy/), segmentação automática de frases, detecção automática de momentos de resposta
|
|
||||||
Exibição de fórmulas, imagens e tabelas | Exibição de fórmulas em formato tex e renderizadas simultaneamente, suporte a fórmulas e destaque de código
|
|
||||||
⭐Plugin AutoGen para vários agentes | [Plugin] Explore a emergência de múltiplos agentes com o AutoGen da Microsoft!
|
|
||||||
Ativar o tema escuro | Adicione ```/?__theme=dark``` ao final da URL para alternar para o tema escuro
|
|
||||||
Suporte a múltiplos modelos LLM | Ser atendido simultaneamente pelo GPT3.5, GPT4, [ChatGLM2](https://github.com/THUDM/ChatGLM2-6B) do Tsinghua University e [MOSS](https://github.com/OpenLMLab/MOSS) da Fudan University se sente incrível, não é mesmo?
|
|
||||||
⭐Modelo de ajuste fino ChatGLM2 | Suporte para carregar o modelo ChatGLM2 ajustado e fornecer plugins de assistência ao ajuste fino do ChatGLM2
|
|
||||||
Mais modelos LLM e suporte para [implantação pela HuggingFace](https://huggingface.co/spaces/qingxu98/gpt-academic) | Integração com a interface Newbing (Bing novo), introdução do [Jittorllms](https://github.com/Jittor/JittorLLMs) da Tsinghua University com suporte a [LLaMA](https://github.com/facebookresearch/llama) e [Panguα](https://openi.org.cn/pangu/)
|
|
||||||
⭐Pacote pip [void-terminal](https://github.com/binary-husky/void-terminal) | Chame todas as funções plugins deste projeto diretamente em Python, sem a GUI (em desenvolvimento)
|
|
||||||
⭐Plugin Terminal do Vácuo | [Plugin] Chame outros plugins deste projeto diretamente usando linguagem natural
|
|
||||||
Apresentação de mais novas funcionalidades (geração de imagens, etc.) ... | Veja no final deste documento ...
|
|
||||||
|
|
||||||
</div>
|
|
||||||
|
|
||||||
|
|
||||||
- Nova interface (altere a opção LAYOUT em `config.py` para alternar entre os "Layouts de lado a lado" e "Layout de cima para baixo")
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://github.com/binary-husky/gpt_academic/assets/96192199/d81137c3-affd-4cd1-bb5e-b15610389762" width="700" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
|
|
||||||
- Todos os botões são gerados dinamicamente através da leitura do `functional.py`, você pode adicionar funcionalidades personalizadas à vontade, liberando sua área de transferência
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/231975334-b4788e91-4887-412f-8b43-2b9c5f41d248.gif" width="700" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
- Aprimoramento/Correção
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/231980294-f374bdcb-3309-4560-b424-38ef39f04ebd.gif" width="700" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
- Se a saída contiver fórmulas, elas serão exibidas tanto em formato tex quanto renderizado para facilitar a cópia e a leitura.
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/230598842-1d7fcddd-815d-40ee-af60-baf488a199df.png" width="700" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
- Não tem vontade de ver o código do projeto? O projeto inteiro está diretamente na boca do chatgpt.
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/226935232-6b6a73ce-8900-4aee-93f9-733c7e6fef53.png" width="700" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
- Combinação de vários modelos de linguagem (ChatGLM + OpenAI-GPT3.5 + [API2D](https://api2d.com/)-GPT4)
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/232537274-deca0563-7aa6-4b5d-94a2-b7c453c47794.png" width="700" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
# Instalação
|
|
||||||
### Método de instalação I: Executar diretamente (Windows, Linux ou MacOS)
|
|
||||||
|
|
||||||
1. Baixe o projeto
|
|
||||||
```sh
|
|
||||||
git clone --depth=1 https://github.com/binary-husky/gpt_academic.git
|
|
||||||
cd gpt_academic
|
|
||||||
```
|
|
||||||
|
|
||||||
2. Configure a API_KEY
|
|
||||||
|
|
||||||
No arquivo `config.py`, configure a API KEY e outras configurações. [Clique aqui para ver o método de configuração em redes especiais](https://github.com/binary-husky/gpt_academic/issues/1). [Página Wiki](https://github.com/binary-husky/gpt_academic/wiki/项目配置说明).
|
|
||||||
|
|
||||||
「 O programa verificará primeiro se existe um arquivo de configuração privada chamado `config_private.py` e substituirá as configurações correspondentes no arquivo `config.py`. Se você entender essa lógica de leitura, é altamente recomendável criar um novo arquivo de configuração chamado `config_private.py` ao lado do `config.py` e copiar as configurações do `config.py` para o `config_private.py` (copiando apenas os itens de configuração que você modificou). 」
|
|
||||||
|
|
||||||
「 Suporte para configurar o projeto por meio de `variáveis de ambiente`, o formato de gravação das variáveis de ambiente pode ser encontrado no arquivo `docker-compose.yml` ou em nossa [página Wiki](https://github.com/binary-husky/gpt_academic/wiki/项目配置说明). A prioridade de leitura das configurações é: `variáveis de ambiente` > `config_private.py` > `config.py`. 」
|
|
||||||
|
|
||||||
|
|
||||||
3. Instale as dependências
|
|
||||||
```sh
|
|
||||||
# (Opção I: Se você está familiarizado com o Python, Python>=3.9) Observação: Use o pip oficial ou o pip da Aliyun. Método temporário para alternar fontes: python -m pip install -r requirements.txt -i https://mirrors.aliyun.com/pypi/simple/
|
|
||||||
python -m pip install -r requirements.txt
|
|
||||||
|
|
||||||
# (Opção II: Use o Anaconda) Os passos também são semelhantes (https://www.bilibili.com/video/BV1rc411W7Dr):
|
|
||||||
conda create -n gptac_venv python=3.11 # Crie um ambiente do Anaconda
|
|
||||||
conda activate gptac_venv # Ative o ambiente do Anaconda
|
|
||||||
python -m pip install -r requirements.txt # Este passo é igual ao da instalação do pip
|
|
||||||
```
|
|
||||||
|
|
||||||
|
|
||||||
<details><summary>Se você quiser suporte para o ChatGLM2 do THU/ MOSS do Fudan/RWKV como backend, clique para expandir</summary>
|
|
||||||
<p>
|
|
||||||
|
|
||||||
[Opcional] Se você quiser suporte para o ChatGLM2 do THU/ MOSS do Fudan, precisará instalar dependências extras (pré-requisitos: familiarizado com o Python + já usou o PyTorch + o computador tem configuração suficiente):
|
|
||||||
```sh
|
|
||||||
# [Opcional Passo I] Suporte para ChatGLM2 do THU. Observações sobre o ChatGLM2 do THU: Se você encontrar o erro "Call ChatGLM fail 不能正常加载ChatGLM的参数" (Falha ao chamar o ChatGLM, não é possível carregar os parâmetros do ChatGLM), consulte o seguinte: 1: A versão instalada por padrão é a versão torch+cpu. Se você quiser usar a versão cuda, desinstale o torch e reinstale uma versão com torch+cuda; 2: Se a sua configuração não for suficiente para carregar o modelo, você pode modificar a precisão do modelo em request_llm/bridge_chatglm.py, alterando todas as ocorrências de AutoTokenizer.from_pretrained("THUDM/chatglm-6b", trust_remote_code=True) para AutoTokenizer.from_pretrained("THUDM/chatglm-6b-int4", trust_remote_code=True)
|
|
||||||
python -m pip install -r request_llms/requirements_chatglm.txt
|
|
||||||
|
|
||||||
# [Opcional Passo II] Suporte para MOSS do Fudan
|
|
||||||
python -m pip install -r request_llms/requirements_moss.txt
|
|
||||||
git clone --depth=1 https://github.com/OpenLMLab/MOSS.git request_llms/moss # Observe que você deve estar no diretório raiz do projeto ao executar este comando
|
|
||||||
|
|
||||||
# [Opcional Passo III] Suporte para RWKV Runner
|
|
||||||
Consulte a página Wiki: https://github.com/binary-husky/gpt_academic/wiki/%E9%80%82%E9%85%8DRWKV-Runner
|
|
||||||
|
|
||||||
# [Opcional Passo IV] Verifique se o arquivo de configuração config.py contém os modelos desejados, os modelos compatíveis são os seguintes (a série jittorllms suporta apenas a solução Docker):
|
|
||||||
AVAIL_LLM_MODELS = ["gpt-3.5-turbo", "api2d-gpt-3.5-turbo", "gpt-4", "api2d-gpt-4", "chatglm", "moss"] # + ["jittorllms_rwkv", "jittorllms_pangualpha", "jittorllms_llama"]
|
|
||||||
```
|
|
||||||
|
|
||||||
</p>
|
|
||||||
</details>
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
4. Execute
|
|
||||||
```sh
|
|
||||||
python main.py
|
|
||||||
```
|
|
||||||
|
|
||||||
### Método de instalação II: Usando o Docker
|
|
||||||
|
|
||||||
0. Implante todas as capacidades do projeto (este é um contêiner grande que inclui CUDA e LaTeX. Não recomendado se você tiver uma conexão lenta com a internet ou pouco espaço em disco)
|
|
||||||
[](https://github.com/binary-husky/gpt_academic/actions/workflows/build-with-all-capacity.yml)
|
|
||||||
|
|
||||||
``` sh
|
|
||||||
# Modifique o arquivo docker-compose.yml para incluir apenas a seção 0 e excluir as outras seções. Em seguida, execute:
|
|
||||||
docker-compose up
|
|
||||||
```
|
|
||||||
|
|
||||||
1. ChatGPT + 文心一言 + spark + outros modelos online (recomendado para a maioria dos usuários)
|
|
||||||
[](https://github.com/binary-husky/gpt_academic/actions/workflows/build-without-local-llms.yml)
|
|
||||||
[](https://github.com/binary-husky/gpt_academic/actions/workflows/build-with-latex.yml)
|
|
||||||
[](https://github.com/binary-husky/gpt_academic/actions/workflows/build-with-audio-assistant.yml)
|
|
||||||
|
|
||||||
``` sh
|
|
||||||
# Modifique o arquivo docker-compose.yml para incluir apenas a seção 1 e excluir as outras seções. Em seguida, execute:
|
|
||||||
docker-compose up
|
|
||||||
```
|
|
||||||
|
|
||||||
Obs.: Se você precisar do plugin Latex, consulte a Wiki. Além disso, você também pode usar a seção 4 ou 0 para obter a funcionalidade do LaTeX.
|
|
||||||
|
|
||||||
2. ChatGPT + ChatGLM2 + MOSS + LLAMA2 + 通义千问 (você precisa estar familiarizado com o [Nvidia Docker](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html#installing-on-ubuntu-and-debian) para executar este modo)
|
|
||||||
[](https://github.com/binary-husky/gpt_academic/actions/workflows/build-with-chatglm.yml)
|
|
||||||
|
|
||||||
``` sh
|
|
||||||
# Modifique o arquivo docker-compose.yml para incluir apenas a seção 2 e excluir as outras seções. Em seguida, execute:
|
|
||||||
docker-compose up
|
|
||||||
```
|
|
||||||
|
|
||||||
|
|
||||||
### Método de instalação III: Outros métodos de implantação
|
|
||||||
1. **Script de execução com um clique para Windows**.
|
|
||||||
Usuários do Windows que não estão familiarizados com o ambiente Python podem baixar o script de execução com um clique da [Release](https://github.com/binary-husky/gpt_academic/releases) para instalar a versão sem modelos locais.
|
|
||||||
A contribuição do script vem de [oobabooga](https://github.com/oobabooga/one-click-installers).
|
|
||||||
|
|
||||||
2. Usar APIs de terceiros, Azure, etc., 文心一言, 星火, consulte a [página Wiki](https://github.com/binary-husky/gpt_academic/wiki/项目配置说明).
|
|
||||||
|
|
||||||
3. Guia para evitar armadilhas na implantação em servidor em nuvem.
|
|
||||||
Consulte o [wiki de implantação em servidor em nuvem](https://github.com/binary-husky/gpt_academic/wiki/%E4%BA%91%E6%9C%8D%E5%8A%A1%E5%99%A8%E8%BF%9C%E7%A8%8B%E9%83%A8%E7%BD%B2%E6%8C%87%E5%8D%97).
|
|
||||||
|
|
||||||
4. Algumas novas plataformas ou métodos de implantação
|
|
||||||
- Use Sealos [implantação com um clique](https://github.com/binary-husky/gpt_academic/issues/993).
|
|
||||||
- Use o WSL2 (Subsistema do Windows para Linux). Consulte [wiki de implantação](https://github.com/binary-husky/gpt_academic/wiki/%E4%BD%BF%E7%94%A8WSL2%EF%BC%88Windows-Subsystem-for-Linux-%E5%AD%90%E7%B3%BB%E7%BB%9F%EF%BC%89%E9%83%A8%E7%BD%B2).
|
|
||||||
- Como executar em um subdiretório da URL (como `http://localhost/subpath`). Consulte [instruções de execução com o FastAPI](docs/WithFastapi.md)
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
# Uso Avançado
|
|
||||||
### I: Personalização de Novos Botões de Atalho (Atalhos Acadêmicos)
|
|
||||||
Abra o arquivo `core_functional.py` em qualquer editor de texto, adicione o seguinte item e reinicie o programa. (Se o botão já existir, o prefixo e o sufixo podem ser modificados a qualquer momento sem reiniciar o programa).
|
|
||||||
Por exemplo:
|
|
||||||
```
|
|
||||||
"超级英译中": {
|
|
||||||
# Prefixo, adicionado antes do seu input. Por exemplo, usado para descrever sua solicitação, como traduzir, explicar o código, revisar, etc.
|
|
||||||
"Prefix": "Por favor, traduza o parágrafo abaixo para o chinês e explique cada termo técnico dentro de uma tabela markdown:\n\n",
|
|
||||||
|
|
||||||
# Sufixo, adicionado após o seu input. Por exemplo, em conjunto com o prefixo, pode-se colocar seu input entre aspas.
|
|
||||||
"Suffix": "",
|
|
||||||
},
|
|
||||||
```
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/226899272-477c2134-ed71-4326-810c-29891fe4a508.png" width="500" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
### II: Personalização de Funções Plugins
|
|
||||||
Crie poderosos plugins de função para executar tarefas que você pode e não pode imaginar.
|
|
||||||
Criar plugins neste projeto é fácil, basta seguir o modelo fornecido, desde que você tenha conhecimento básico de Python.
|
|
||||||
Consulte o [Guia dos Plugins de Função](https://github.com/binary-husky/gpt_academic/wiki/%E5%87%BD%E6%95%B0%E6%8F%92%E4%BB%B6%E6%8C%87%E5%8D%97) para mais detalhes.
|
|
||||||
|
|
||||||
|
|
||||||
# Atualizações
|
|
||||||
### I: Dinâmico
|
|
||||||
|
|
||||||
1. Função de salvar conversas. Chame a função "Salvar a conversa atual" na área de plugins para salvar a conversa atual em um arquivo HTML legível e recuperável. Além disso, chame a função "Carregar histórico de conversas" na área de plugins (menu suspenso) para restaurar conversas anteriores.
|
|
||||||
Dica: Se você clicar diretamente em "Carregar histórico de conversas" sem especificar o arquivo, poderá visualizar o cache do histórico do arquivo HTML.
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/235222390-24a9acc0-680f-49f5-bc81-2f3161f1e049.png" width="500" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
2. ⭐Tradução de artigos Latex/Arxiv⭐
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://github.com/binary-husky/gpt_academic/assets/96192199/002a1a75-ace0-4e6a-94e2-ec1406a746f1" height="250" > ===>
|
|
||||||
<img src="https://github.com/binary-husky/gpt_academic/assets/96192199/9fdcc391-f823-464f-9322-f8719677043b" height="250" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
3. Terminal vazio (entendendo a intenção do usuário a partir do texto em linguagem natural e chamando automaticamente outros plugins)
|
|
||||||
|
|
||||||
- Passo 1: Digite "Por favor, chame o plugin 'Traduzir artigo PDF' e forneça o link https://openreview.net/pdf?id=rJl0r3R9KX"
|
|
||||||
- Passo 2: Clique em "Terminal vazio"
|
|
||||||
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://github.com/binary-husky/gpt_academic/assets/96192199/66f1b044-e9ff-4eed-9126-5d4f3668f1ed" width="500" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
4. Design de recursos modular, interface simples com suporte a recursos poderosos
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/229288270-093643c1-0018-487a-81e6-1d7809b6e90f.png" height="400" >
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/227504931-19955f78-45cd-4d1c-adac-e71e50957915.png" height="400" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
5. Tradução e interpretação de outros projetos de código aberto
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/226935232-6b6a73ce-8900-4aee-93f9-733c7e6fef53.png" height="250" >
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/226969067-968a27c1-1b9c-486b-8b81-ab2de8d3f88a.png" height="250" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
6. Recursos adicionais para [live2d](https://github.com/fghrsh/live2d_demo) (desativados por padrão, requer modificação no arquivo `config.py`)
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/236432361-67739153-73e8-43fe-8111-b61296edabd9.png" width="500" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
7. Geração de imagens pela OpenAI
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://github.com/binary-husky/gpt_academic/assets/96192199/bc7ab234-ad90-48a0-8d62-f703d9e74665" width="500" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
8. Análise e resumo de áudio pela OpenAI
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://github.com/binary-husky/gpt_academic/assets/96192199/709ccf95-3aee-498a-934a-e1c22d3d5d5b" width="500" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
9. Correção de erros em texto e código LaTeX
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://github.com/binary-husky/gpt_academic/assets/96192199/651ccd98-02c9-4464-91e1-77a6b7d1b033" height="200" > ===>
|
|
||||||
<img src="https://github.com/binary-husky/gpt_academic/assets/96192199/476f66d9-7716-4537-b5c1-735372c25adb" height="200">
|
|
||||||
</div>
|
|
||||||
|
|
||||||
10. Alternância de idioma e tema
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://github.com/binary-husky/gpt_academic/assets/96192199/b6799499-b6fb-4f0c-9c8e-1b441872f4e8" width="500" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
### II: Versões:
|
|
||||||
- Versão 3.70 (a fazer): Melhorar o plugin AutoGen e projetar uma série de plugins relacionados.
|
|
||||||
- Versão 3.60: Introdução do AutoGen como base para a próxima geração de plugins.
|
|
||||||
- Versão 3.57: Suporte para GLM3, Starfire v3, Wenxin Yiyan v4, correção de bugs relacionados a modelos locais executados simultaneamente.
|
|
||||||
- Versão 3.56: Suporte para adicionar dinamicamente botões de função básicos e nova página de resumo em PDF.
|
|
||||||
- Versão 3.55: Reformulação da interface do usuário, introdução de janelas flutuantes e menus.
|
|
||||||
- Versão 3.54: Novo interpretador de código dinâmico (Code Interpreter) (em desenvolvimento)
|
|
||||||
- Versão 3.53: Suporte para alterar dinamicamente o tema da interface, melhorias de estabilidade e correção de conflitos entre vários usuários.
|
|
||||||
- Versão 3.50: Chamada de todas as funções de plugins deste projeto usando linguagem natural (Terminal vazio), suporte a categorização de plugins, melhorias na interface do usuário e design de novos temas.
|
|
||||||
- Versão 3.49: Suporte para Baidu Qianfan Platform e Wenxin Yiyan.
|
|
||||||
- Versão 3.48: Suporte para Alibaba DAMO Academy Tongyi Qianwen, Shanghai AI-Lab Shusheng e Xunfei Xinghuo.
|
|
||||||
- Versão 3.46: Suporte para diálogos em tempo real totalmente automáticos.
|
|
||||||
- Versão 3.45: Suporte para personalização do modelo ChatGLM2.
|
|
||||||
- Versão 3.44: Suporte oficial ao Azure, aprimoramentos na usabilidade da interface.
|
|
||||||
- Versão 3.4: Tradução completa de artigos Arxiv/Latex, correção de artigos Latex.
|
|
||||||
- Versão 3.3: Funcionalidade de consulta a informações na internet.
|
|
||||||
- Versão 3.2: Maior suporte para parâmetros de função de plugins (função de salvar conversas, interpretação de código em qualquer linguagem + perguntas sobre combinações LLM arbitrariamente).
|
|
||||||
- Versão 3.1: Suporte para fazer perguntas a modelos GPT múltiplos! Suporte para API2D, balanceamento de carga em vários APIKeys.
|
|
||||||
- Versão 3.0: Suporte para chatglm e outros pequenos modelos LLM.
|
|
||||||
- Versão 2.6: Refatoração da estrutura de plugins, melhoria na interação, adição de mais plugins.
|
|
||||||
- Versão 2.5: Auto-atualizável, resolve problemas de texto muito longo ou estouro de tokens ao resumir grandes projetos de código.
|
|
||||||
- Versão 2.4: (1) Novo recurso de tradução completa de PDF; (2) Nova função para alternar a posição da área de input; (3) Nova opção de layout vertical; (4) Melhoria dos plugins de função em várias threads.
|
|
||||||
- Versão 2.3: Melhorias na interação em várias threads.
|
|
||||||
- Versão 2.2: Suporte para recarregar plugins sem reiniciar o programa.
|
|
||||||
- Versão 2.1: Layout dobrável.
|
|
||||||
- Versão 2.0: Introdução de plugins de função modular.
|
|
||||||
- Versão 1.0: Funcionalidades básicas.
|
|
||||||
|
|
||||||
GPT Academic QQ Group: `610599535`
|
|
||||||
|
|
||||||
- Problemas conhecidos
|
|
||||||
- Alguns plugins de tradução de navegadores podem interferir na execução deste software.
|
|
||||||
- A biblioteca Gradio possui alguns bugs de compatibilidade conhecidos. Certifique-se de instalar o Gradio usando o arquivo `requirement.txt`.
|
|
||||||
|
|
||||||
### III: Temas
|
|
||||||
Você pode alterar o tema atualizando a opção `THEME` (config.py).
|
|
||||||
1. `Chuanhu-Small-and-Beautiful` [Link](https://github.com/GaiZhenbiao/ChuanhuChatGPT/)
|
|
||||||
|
|
||||||
|
|
||||||
### IV: Branches de Desenvolvimento deste Projeto
|
|
||||||
|
|
||||||
1. Branch `master`: Branch principal, versão estável.
|
|
||||||
2. Branch `frontier`: Branch de desenvolvimento, versão de teste.
|
|
||||||
|
|
||||||
|
|
||||||
### V: Referências para Aprendizado
|
|
||||||
|
|
||||||
```
|
|
||||||
O código referenciou muitos projetos excelentes, em ordem aleatória:
|
|
||||||
|
|
||||||
# Tsinghua ChatGLM2-6B:
|
|
||||||
https://github.com/THUDM/ChatGLM2-6B
|
|
||||||
|
|
||||||
# Tsinghua JittorLLMs:
|
|
||||||
https://github.com/Jittor/JittorLLMs
|
|
||||||
|
|
||||||
# ChatPaper:
|
|
||||||
https://github.com/kaixindelele/ChatPaper
|
|
||||||
|
|
||||||
# Edge-GPT:
|
|
||||||
https://github.com/acheong08/EdgeGPT
|
|
||||||
|
|
||||||
# ChuanhuChatGPT:
|
|
||||||
https://github.com/GaiZhenbiao/ChuanhuChatGPT
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
# Oobabooga instalador com um clique:
|
|
||||||
https://github.com/oobabooga/instaladores-de-um-clique
|
|
||||||
|
|
||||||
# Mais:
|
|
||||||
https://github.com/gradio-app/gradio
|
|
||||||
https://github.com/fghrsh/live2d_demo
|
|
||||||
@@ -1,360 +0,0 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
> **Примечание**
|
|
||||||
>
|
|
||||||
> Этот README был переведен с помощью GPT (реализовано с помощью плагина этого проекта) и не может быть полностью надежным, пожалуйста, внимательно проверьте результаты перевода.
|
|
||||||
>
|
|
||||||
> 7 ноября 2023 года: При установке зависимостей, пожалуйста, выберите **указанные версии** из `requirements.txt`. Команда установки: `pip install -r requirements.txt`.
|
|
||||||
|
|
||||||
|
|
||||||
# <div align=center><img src="logo.png" width="40"> GPT Academic (GPT Академический)</div>
|
|
||||||
|
|
||||||
**Если вам нравится этот проект, пожалуйста, поставьте звезду; если у вас есть удобные горячие клавиши или плагины, приветствуются pull requests!**
|
|
||||||
Чтобы перевести этот проект на произвольный язык с помощью GPT, прочтите и выполните [`multi_language.py`](multi_language.py) (экспериментально).
|
|
||||||
|
|
||||||
> **Примечание**
|
|
||||||
>
|
|
||||||
> 1. Пожалуйста, обратите внимание, что только плагины (кнопки), выделенные **жирным шрифтом**, поддерживают чтение файлов, некоторые плагины находятся в выпадающем меню **плагинов**. Кроме того, мы с радостью приветствуем и обрабатываем PR для любых новых плагинов с **наивысшим приоритетом**.
|
|
||||||
>
|
|
||||||
> 2. Функции каждого файла в этом проекте подробно описаны в [отчете о самостоятельном анализе проекта `self_analysis.md`](https://github.com/binary-husky/gpt_academic/wiki/GPT‐Academic项目自译解报告). С каждым новым релизом вы также можете в любое время нажать на соответствующий функциональный плагин, вызвать GPT для повторной генерации сводного отчета о самоанализе проекта. Часто задаваемые вопросы [`wiki`](https://github.com/binary-husky/gpt_academic/wiki) | [обычные методы установки](#installation) | [скрипт одношаговой установки](https://github.com/binary-husky/gpt_academic/releases) | [инструкции по настройке](https://github.com/binary-husky/gpt_academic/wiki/项目配置说明).
|
|
||||||
>
|
|
||||||
> 3. Этот проект совместим и настоятельно рекомендуется использование китайской NLP-модели ChatGLM и других моделей больших языков производства Китая. Поддерживает одновременное использование нескольких ключей API, которые можно указать в конфигурационном файле, например, `API_KEY="openai-key1,openai-key2,azure-key3,api2d-key4"`. Если нужно временно заменить `API_KEY`, введите временный `API_KEY` в окне ввода и нажмите Enter для его подтверждения.
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
<div align="center">
|
|
||||||
|
|
||||||
Функции (⭐= Недавно добавленные функции) | Описание
|
|
||||||
--- | ---
|
|
||||||
⭐[Подключение новой модели](https://github.com/binary-husky/gpt_academic/wiki/%E5%A6%82%E4%BD%95%E5%88%87%E6%8D%A2%E6%A8%A1%E5%9E%8B)！ | Baidu [QianFan](https://cloud.baidu.com/doc/WENXINWORKSHOP/s/Nlks5zkzu) и WenxinYiYan, [TongYiQianWen](https://modelscope.cn/models/qwen/Qwen-7B-Chat/summary), Shanghai AI-Lab [ShuSheng](https://github.com/InternLM/InternLM), Xunfei [XingHuo](https://xinghuo.xfyun.cn/), [LLaMa2](https://huggingface.co/meta-llama/Llama-2-7b-chat-hf), ZhiPu API, DALLE3
|
|
||||||
Улучшение, перевод, объяснение кода | Одним нажатием выполнить поиск синтаксических ошибок в научных статьях, переводить, объяснять код
|
|
||||||
[Настройка горячих клавиш](https://www.bilibili.com/video/BV14s4y1E7jN) | Поддержка настройки горячих клавиш
|
|
||||||
Модульный дизайн | Поддержка настраиваемых мощных [плагинов](https://github.com/binary-husky/gpt_academic/tree/master/crazy_functions), плагины поддерживают [горячую замену](https://github.com/binary-husky/gpt_academic/wiki/%E5%87%BD%E6%95%B0%E6%8F%92%E4%BB%B6%E6%8C%87%E5%8D%97)
|
|
||||||
[Профилирование кода](https://www.bilibili.com/video/BV1cj411A7VW) | [Плагин] Одним нажатием можно профилировать дерево проекта Python/C/C++/Java/Lua/... или [проанализировать самого себя](https://www.bilibili.com/video/BV1cj411A7VW)
|
|
||||||
Просмотр статей, перевод статей | [Плагин] Одним нажатием прочитать полный текст статьи в формате LaTeX/PDF и сгенерировать аннотацию
|
|
||||||
Перевод LaTeX статей, [улучшение](https://www.bilibili.com/video/BV1FT411H7c5/)| [Плагин] Одним нажатием перевести или улучшить статьи в формате LaTeX
|
|
||||||
Генерация пакетного комментария | [Плагин] Одним нажатием сгенерировать многострочный комментарий к функции
|
|
||||||
Перевод Markdown на английский и китайский | [Плагин] Вы видели документацию на сверху на пяти языках? [README](https://github.com/binary-husky/gpt_academic/blob/master/docs/README_EN.md)`
|
|
||||||
Анализ и создание отчета в формате чата | [Плагин] Автоматически генерируйте сводный отчет после выполнения
|
|
||||||
Функция перевода полноценной PDF статьи | [Плагин] Изъять название и аннотацию статьи из PDF + переводить полный текст (многопоточно)
|
|
||||||
[Arxiv помощник](https://www.bilibili.com/video/BV1LM4y1279X) | [Плагин] Просто введите URL статьи на arXiv, чтобы одним нажатием выполнить перевод аннотации + загрузить PDF
|
|
||||||
Одним кликом проверить статью на LaTeX | [Плагин] Проверка грамматики и правописания статьи LaTeX, добавление PDF в качестве справки
|
|
||||||
[Помощник Google Scholar](https://www.bilibili.com/video/BV19L411U7ia) | [Плагин] Создайте "related works" с помощью Google Scholar URL по вашему выбору.
|
|
||||||
Агрегирование интернет-информации + GPT | [Плагин] [GPT получает информацию из интернета](https://www.bilibili.com/video/BV1om4y127ck) и отвечает на вопросы, чтобы информация никогда не устаревала
|
|
||||||
⭐Точный перевод статей Arxiv ([Docker](https://github.com/binary-husky/gpt_academic/pkgs/container/gpt_academic_with_latex)) | [Плагин] [Переводите статьи Arxiv наивысшего качества](https://www.bilibili.com/video/BV1dz4y1v77A/) всего одним нажатием. Сейчас это лучший инструмент для перевода научных статей
|
|
||||||
⭐[Реальное время ввода голосом](https://github.com/binary-husky/gpt_academic/blob/master/docs/use_audio.md) | [Плагин] Асинхронно [слушать аудио](https://www.bilibili.com/video/BV1AV4y187Uy/), автоматически разбивать на предложения, автоматически находить момент для ответа
|
|
||||||
Отображение формул/изображений/таблиц | Поддержка отображения формул в форме [tex и рендеринга](https://user-images.githubusercontent.com/96192199/230598842-1d7fcddd-815d-40ee-af60-baf488a199df.png), поддержка подсветки синтаксиса формул и кода
|
|
||||||
⭐Плагин AutoGen для множества интеллектуальных агентов | [Плагин] Используйте Microsoft AutoGen для исследования возможностей интеллектуального всплытия нескольких агентов!
|
|
||||||
Запуск [темной темы](https://github.com/binary-husky/gpt_academic/issues/173) | Добавьте `/?__theme=dark` в конец URL в браузере, чтобы переключиться на темную тему
|
|
||||||
[Поддержка нескольких моделей LLM](https://www.bilibili.com/video/BV1wT411p7yf) | Быть обслуживаемым GPT3.5, GPT4, [ChatGLM2 из Цинхуа](https://github.com/THUDM/ChatGLM2-6B), [MOSS из Фуданя](https://github.com/OpenLMLab/MOSS) одновременно должно быть очень приятно, не так ли?
|
|
||||||
⭐Модель ChatGLM2 Fine-tune | Поддержка загрузки модели ChatGLM2 Fine-tune, предоставляет вспомогательный плагин ChatGLM2 Fine-tune
|
|
||||||
Больше моделей LLM, поддержка [развертывания huggingface](https://huggingface.co/spaces/qingxu98/gpt-academic) | Включение интерфейса Newbing (новый Bing), введение поддержки китайских [Jittorllms](https://github.com/Jittor/JittorLLMs) для поддержки [LLaMA](https://github.com/facebookresearch/llama) и [Panguα](https://openi.org.cn/pangu/)
|
|
||||||
⭐Пакет pip [void-terminal](https://github.com/binary-husky/void-terminal) | Без GUI вызывайте все функциональные плагины этого проекта прямо из Python (разрабатывается)
|
|
||||||
⭐Плагин пустого терминала | [Плагин] Используя естественный язык, напрямую распоряжайтесь другими плагинами этого проекта
|
|
||||||
Больше новых функций (генерация изображений и т. д.) ... | Смотрите в конце этого документа ...
|
|
||||||
</div>
|
|
||||||
|
|
||||||
|
|
||||||
- Новый интерфейс (изменение опции LAYOUT в `config.py` позволяет переключиться между "расположением слева и справа" и "расположением сверху и снизу")
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://github.com/binary-husky/gpt_academic/assets/96192199/d81137c3-affd-4cd1-bb5e-b15610389762" width="700" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
|
|
||||||
- Все кнопки генерируются динамически на основе `functional.py` и могут быть свободно дополнены, освобождая буфер обмена
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/231975334-b4788e91-4887-412f-8b43-2b9c5f41d248.gif" width="700" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
- Улучшение/исправление
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/231980294-f374bdcb-3309-4560-b424-38ef39f04ebd.gif" width="700" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
- Если вывод содержит формулы, они отображаются одновременно в виде tex и отрендеренного вида для удобства копирования и чтения
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/230598842-1d7fcddd-815d-40ee-af60-baf488a199df.png" width="700" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
- Не хочешь смотреть код проекта? Весь проект сразу в уста ChatGPT
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/226935232-6b6a73ce-8900-4aee-93f9-733c7e6fef53.png" width="700" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
- Смешанное использование нескольких больших языковых моделей (ChatGLM + OpenAI-GPT3.5 + [API2D](https://api2d.com/)-GPT4)
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/232537274-deca0563-7aa6-4b5d-94a2-b7c453c47794.png" width="700" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
# Установка
|
|
||||||
### Метод установки I: Прямой запуск (Windows, Linux или MacOS)
|
|
||||||
|
|
||||||
1. Скачайте проект
|
|
||||||
```sh
|
|
||||||
git clone --depth=1 https://github.com/binary-husky/gpt_academic.git
|
|
||||||
cd gpt_academic
|
|
||||||
```
|
|
||||||
|
|
||||||
2. Настройте API_KEY
|
|
||||||
|
|
||||||
В файле `config.py` настройте API KEY и другие настройки, [нажмите здесь, чтобы узнать способы настройки в специальных сетевых средах](https://github.com/binary-husky/gpt_academic/issues/1). [Инструкции по настройке проекта](https://github.com/binary-husky/gpt_academic/wiki/Сonfig-Instructions).
|
|
||||||
|
|
||||||
「 Программа будет в первую очередь проверять наличие файла config_private.py с приватными настройками и заменять соответствующие настройки в файле config.py на те, которые указаны в файле config_private.py. Если вы понимаете эту логику, мы настоятельно рекомендуем вам создать новый файл настроек config_private.py рядом с файлом config.py и скопировать туда настройки из config.py (только те, которые вы изменяли). 」
|
|
||||||
|
|
||||||
「 Поддерживается настроить проект с помощью `переменных среды`. Пример настройки переменных среды можно найти в файле docker-compose.yml или на нашей [странице вики](https://github.com/binary-husky/gpt_academic/wiki/Сonfig-Instructions). Приоритет настроек: `переменные среды` > `config_private.py` > `config.py`. 」
|
|
||||||
|
|
||||||
|
|
||||||
3. Установите зависимости
|
|
||||||
```sh
|
|
||||||
# (Выбор I: Если знакомы с Python, python>=3.9). Примечание: используйте официальный pip-репозиторий или пакетный репозиторий Alibaba, временный способ изменить источник: python -m pip install -r requirements.txt -i https://mirrors.aliyun.com/pypi/simple/
|
|
||||||
python -m pip install -r requirements.txt
|
|
||||||
|
|
||||||
# (Выбор II: Используйте Anaconda). Шаги аналогичны (https://www.bilibili.com/video/BV1rc411W7Dr)：
|
|
||||||
conda create -n gptac_venv python=3.11 # Создание среды Anaconda
|
|
||||||
conda activate gptac_venv # Активация среды Anaconda
|
|
||||||
python -m pip install -r requirements.txt # Здесь все тоже самое, что и с установкой для pip
|
|
||||||
```
|
|
||||||
|
|
||||||
|
|
||||||
<details><summary>Если вам нужна поддержка ChatGLM2 от Цинхуа/MOSS от Фуданя/Раннера RWKV как бэкенда, нажмите, чтобы развернуть</summary>
|
|
||||||
<p>
|
|
||||||
|
|
||||||
【Опциональный шаг】Если вам нужна поддержка ChatGLM2 от Цинхуа/Сервиса MOSS от Фуданя, вам понадобится дополнительно установить дополнительные зависимости (предполагается, что вы знакомы с Python + PyTorch + у вас достаточно мощный компьютер):
|
|
||||||
```sh
|
|
||||||
# 【Опциональный шаг I】Поддержка ChatGLM2 от Цинхуа. Примечание к ChatGLM от Цинхуа: Если вы столкнулись с ошибкой "Call ChatGLM fail 不能正常加载ChatGLM的参数", обратите внимание на следующее: 1: По умолчанию установлена версия torch+cpu, для использования cuda необходимо удалить torch и установить версию torch+cuda; 2: Если вы не можете загрузить модель из-за недостаточной мощности компьютера, вы можете изменить точность модели в файле request_llm/bridge_chatglm.py, заменив AutoTokenizer.from_pretrained("THUDM/chatglm-6b", trust_remote_code=True) на AutoTokenizer.from_pretrained("THUDM/chatglm-6b-int4", trust_remote_code=True)
|
|
||||||
python -m pip install -r request_llms/requirements_chatglm.txt
|
|
||||||
|
|
||||||
# 【Опциональный шаг II】Поддержка MOSS от Фуданя
|
|
||||||
python -m pip install -r request_llms/requirements_moss.txt
|
|
||||||
git clone --depth=1 https://github.com/OpenLMLab/MOSS.git request_llms/moss # Обратите внимание, что когда вы запускаете эту команду, вы должны находиться в корневой папке проекта
|
|
||||||
|
|
||||||
# 【Опциональный шаг III】Поддержка RWKV Runner
|
|
||||||
Смотрите вики: https://github.com/binary-husky/gpt_academic/wiki/Поддержка-RWKV-Runner
|
|
||||||
|
|
||||||
# 【Опциональный шаг IV】Убедитесь, что config.py содержит все нужные вам модели. Пример:
|
|
||||||
AVAIL_LLM_MODELS = ["gpt-3.5-turbo", "api2d-gpt-3.5-turbo", "gpt-4", "api2d-gpt-4", "chatglm", "moss"] # + ["jittorllms_rwkv", "jittorllms_pangualpha", "jittorllms_llama"]
|
|
||||||
```
|
|
||||||
|
|
||||||
</p>
|
|
||||||
</details>
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
4. Запустите программу
|
|
||||||
```sh
|
|
||||||
python main.py
|
|
||||||
```
|
|
||||||
|
|
||||||
### Метод установки II: Используйте Docker
|
|
||||||
|
|
||||||
0. Установка всех возможностей проекта (это большой образ с поддержкой cuda и LaTeX; но если у вас медленный интернет или маленький жесткий диск, мы не рекомендуем использовать этот метод).
|
|
||||||
[](https://github.com/binary-husky/gpt_academic/actions/workflows/build-with-all-capacity.yml)
|
|
||||||
|
|
||||||
``` sh
|
|
||||||
# Измените файл docker-compose.yml, сохраните метод 0 и удалите другие методы. Затем запустите:
|
|
||||||
docker-compose up
|
|
||||||
```
|
|
||||||
|
|
||||||
1. Чат GPT + 文心一言 + Spark и другие онлайн-модели (рекомендуется для большинства пользователей)
|
|
||||||
[](https://github.com/binary-husky/gpt_academic/actions/workflows/build-without-local-llms.yml)
|
|
||||||
[](https://github.com/binary-husky/gpt_academic/actions/workflows/build-with-latex.yml)
|
|
||||||
[](https://github.com/binary-husky/gpt_academic/actions/workflows/build-with-audio-assistant.yml)
|
|
||||||
|
|
||||||
``` sh
|
|
||||||
# Измените файл docker-compose.yml, сохраните метод 1 и удалите другие методы. Затем запустите:
|
|
||||||
docker-compose up
|
|
||||||
```
|
|
||||||
|
|
||||||
P.S. Если вам нужен функционал, связанный с LaTeX, обратитесь к разделу Wiki. Кроме того, вы также можете использовать схему 4 или схему 0 для доступа к функционалу LaTeX.
|
|
||||||
|
|
||||||
2. Чат GPT + ChatGLM2 + MOSS + LLAMA2 + TakyiQ & Другие попытки ввести в обиход
|
|
||||||
[](https://github.com/binary-husky/gpt_academic/actions/workflows/build-with-chatglm.yml)
|
|
||||||
|
|
||||||
``` sh
|
|
||||||
# Измените файл docker-compose.yml, сохраните метод 2 и удалите другие методы. Затем запустите:
|
|
||||||
docker-compose up
|
|
||||||
```
|
|
||||||
|
|
||||||
|
|
||||||
### Метод установки III: Другие способы развертывания
|
|
||||||
1. **Скрипты запуска одним нажатием для Windows**.
|
|
||||||
Пользователи Windows, не знакомые с окружением Python, могут загрузить одну из версий в разделе [Релизы](https://github.com/binary-husky/gpt_academic/releases) для установки версии без локальных моделей.
|
|
||||||
Скрипты взяты из вкладки [oobabooga](https://github.com/oobabooga/one-click-installers).
|
|
||||||
|
|
||||||
2. Использование сторонних API, Azure и т. д., см. страницу [вики](https://github.com/binary-husky/gpt_academic/wiki/Сonfig-Instructions)
|
|
||||||
|
|
||||||
3. Руководство по развертыванию на удаленном сервере.
|
|
||||||
Пожалуйста, посетите [вики-страницу развертывания на облачном сервере](https://github.com/binary-husky/gpt_academic/wiki/Руководство-по-развертыванию-на-облаке).
|
|
||||||
|
|
||||||
4. Некоторые новые платформы или методы развертывания
|
|
||||||
- Использование Sealos [для однократного развертывания](https://github.com/binary-husky/gpt_academic/issues/993)
|
|
||||||
- Использование WSL2 (Windows Subsystem for Linux). См. [Руководство развертывания-2](https://github.com/binary-husky/gpt_academic/wiki/Using-WSL2-for-deployment)
|
|
||||||
- Как запустить на вложенном URL-адресе (например, `http://localhost/subpath`). См. [Инструкции по работе с FastAPI](docs/WithFastapi.md)
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
# Расширенное использование
|
|
||||||
### I: Пользовательские удобные кнопки (академические сочетания клавиш)
|
|
||||||
Откройте файл `core_functional.py` в любом текстовом редакторе и добавьте следующие записи, затем перезапустите программу. (Если кнопка уже существует, то префикс и суффикс поддерживают горячую замену без перезапуска программы.)
|
|
||||||
Например,
|
|
||||||
```
|
|
||||||
"Супер-англо-русский перевод": {
|
|
||||||
# Префикс, который будет добавлен перед вашим вводом. Например, используется для описания вашего запроса, например, перевода, объяснения кода, редактирования и т.д.
|
|
||||||
"Префикс": "Пожалуйста, переведите следующий абзац на русский язык, а затем покажите каждый термин на экране с помощью таблицы Markdown:\n\n",
|
|
||||||
|
|
||||||
# Суффикс, который будет добавлен после вашего ввода. Например, можно использовать с префиксом, чтобы заключить ваш ввод в кавычки.
|
|
||||||
"Суффикс": "",
|
|
||||||
},
|
|
||||||
```
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/226899272-477c2134-ed71-4326-810c-29891fe4a508.png" width="500" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
### II: Пользовательские функциональные плагины
|
|
||||||
Создавайте мощные функциональные плагины для выполнения любых задач, которые вам нужны и которых вы и не можете себе представить.
|
|
||||||
Создание плагина для этого проекта и его отладка являются простыми задачами, и если у вас есть базовые знания Python, вы можете реализовать свой собственный функциональный плагин, используя наши предоставленные шаблоны.
|
|
||||||
Дополнительную информацию см. в [Руководстве по функциональным плагинам](https://github.com/binary-husky/gpt_academic/wiki/%E5%87%BD%E6%95%B0%E6%8F%92%E4%BB%B6%E6%8C%87%E5%8D%97).
|
|
||||||
|
|
||||||
|
|
||||||
# Обновления
|
|
||||||
### I: Динамические
|
|
||||||
|
|
||||||
1. Функция сохранения диалога. Вызовите "Сохранить текущий диалог" в области функциональных плагинов, чтобы сохранить текущий диалог в виде читаемого и восстанавливаемого html-файла.
|
|
||||||
Кроме того, можно использовать "Загрузить архивный файл диалога" в области функциональных плагинов (выпадающее меню), чтобы восстановить предыдущий разговор.
|
|
||||||
Подсказка: если не указывать файл и просто щелкнуть "Загрузить архивный файл диалога", можно просмотреть кэш сохраненных html-архивов.
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/235222390-24a9acc0-680f-49f5-bc81-2f3161f1e049.png" width="500" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
2. ⭐Перевод Latex/Arxiv статей⭐
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://github.com/binary-husky/gpt_academic/assets/96192199/002a1a75-ace0-4e6a-94e2-ec1406a746f1" height="250" > ===>
|
|
||||||
<img src="https://github.com/binary-husky/gpt_academic/assets/96192199/9fdcc391-f823-464f-9322-f8719677043b" height="250" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
3. Void Terminal (понимание пользовательских намерений из естественного языка и автоматическое вызов других плагинов)
|
|
||||||
|
|
||||||
- Шаг 1: Введите "Пожалуйста, вызовите плагин для перевода PDF-статьи, адрес которой https://openreview.net/pdf?id=rJl0r3R9KX".
|
|
||||||
- Шаг 2: Нажмите "Void Terminal".
|
|
||||||
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://github.com/binary-husky/gpt_academic/assets/96192199/66f1b044-e9ff-4eed-9126-5d4f3668f1ed" width="500" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
4. Модульный дизайн функционала, позволяющий реализовать мощные функции с помощью простых интерфейсов
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/229288270-093643c1-0018-487a-81e6-1d7809b6e90f.png" height="400" >
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/227504931-19955f78-45cd-4d1c-adac-e71e50957915.png" height="400" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
5. Перевод и анализ других открытых проектов
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/226935232-6b6a73ce-8900-4aee-93f9-733c7e6fef53.png" height="250" >
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/226969067-968a27c1-1b9c-486b-8b81-ab2de8d3f88a.png" height="250" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
6. Функциональность для украшения[meme](https://github.com/fghrsh/live2d_demo) (по умолчанию отключена, требуется изменение файла `config.py`)
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://user-images.githubusercontent.com/96192199/236432361-67739153-73e8-43fe-8111-b61296edabd9.png" width="500" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
7. Генерация изображений с помощью OpenAI
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://github.com/binary-husky/gpt_academic/assets/96192199/bc7ab234-ad90-48a0-8d62-f703d9e74665" width="500" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
8. Анализ и обобщение аудио с помощью OpenAI
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://github.com/binary-husky/gpt_academic/assets/96192199/709ccf95-3aee-498a-934a-e1c22d3d5d5b" width="500" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
9. Проверка и исправление ошибок во всем тексте LaTeX
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://github.com/binary-husky/gpt_academic/assets/96192199/651ccd98-02c9-4464-91e1-77a6b7d1b033" height="200" > ===>
|
|
||||||
<img src="https://github.com/binary-husky/gpt_academic/assets/96192199/476f66d9-7716-4537-b5c1-735372c25adb" height="200">
|
|
||||||
</div>
|
|
||||||
|
|
||||||
10. Изменение языка и темы
|
|
||||||
<div align="center">
|
|
||||||
<img src="https://github.com/binary-husky/gpt_academic/assets/96192199/b6799499-b6fb-4f0c-9c8e-1b441872f4e8" width="500" >
|
|
||||||
</div>
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
### II: Версии:
|
|
||||||
- Версия 3.70 (в планах): Оптимизация темы AutoGen и разработка ряда дополнительных плагинов
|
|
||||||
- Версия 3.60: Внедрение AutoGen в качестве фундамента нового поколения плагинов
|
|
||||||
- Версия 3.57: Поддержка GLM3, Starfire v3, Wenxin One Word v4, исправление ошибок при совместном использовании локальной модели
|
|
||||||
- Версия 3.56: Поддержка добавления дополнительных функциональных кнопок в реальном времени, новая страница отчетов в формате PDF
|
|
||||||
- Версия 3.55: Переработка пользовательского интерфейса, внедрение плавающего окна и панели меню
|
|
||||||
- Версия 3.54: Добавлен интерпретатор кода (Code Interpreter) (в разработке)
|
|
||||||
- Версия 3.53: Динамический выбор различных тем интерфейса, повышение стабильности и решение проблемы конфликтов между несколькими пользователями
|
|
||||||
- Версия 3.50: Использование естественного языка для вызова всех функциональных плагинов проекта (Void Terminal), поддержка категоризации плагинов, улучшение пользовательского интерфейса, разработка новых тем
|
|
||||||
- Версия 3.49: Поддержка платформы Baidu Qianfan и Wenxin One Word
|
|
||||||
- Версия 3.48: Поддержка Ali Dharma Institute, Shanghai AI-Lab Scholar, Xunfei Starfire
|
|
||||||
- Версия 3.46: Поддержка реального голосового диалога с полной автоматизацией
|
|
||||||
- Версия 3.45: Поддержка настраиваемой модели ChatGLM2
|
|
||||||
- Версия 3.44: Официальная поддержка Azure, улучшение удобства пользовательского интерфейса
|
|
||||||
- Версия 3.4: +Перевод полных текстов PDF, +корректировка латексных документов
|
|
||||||
- Версия 3.3: +Интернет-информационные функции
|
|
||||||
- Версия 3.2: Поддержка дополнительных параметров в функциональных плагинах (функция сохранения диалога, интерпретация кода на любом языке + одновременный вопрос о любом комбинированном LLM)
|
|
||||||
- Версия 3.1: Поддержка одновременного обращения к нескольким моделям gpt! Поддержка API2D, поддержка равномерной нагрузки нескольких api-ключей
|
|
||||||
- Версия 3.0: Поддержка chatglm и других небольших моделей llm
|
|
||||||
- Версия 2.6: Переработка структуры плагинов для повышения интерактивности, добавление дополнительных плагинов
|
|
||||||
- Версия 2.5: Автоматическое обновление, решение проблемы с длиной текста и переполнением токенов при обработке текста
|
|
||||||
- Версия 2.4: (1) Добавление функции полного перевода PDF; (2) Добавление функции изменения позиции объекта ввода; (3) Добавление функции вертикального размещения; (4) Оптимизация многопоточных функциональных плагинов.
|
|
||||||
- Версия 2.3: Улучшение интерактивности многопоточности
|
|
||||||
- Версия 2.2: Поддержка живой перезагрузки функциональных плагинов
|
|
||||||
- Версия 2.1: Складываемый макет
|
|
||||||
- Версия 2.0: Введение модульных функциональных плагинов
|
|
||||||
- Версия 1.0: Базовые функции
|
|
||||||
|
|
||||||
GPT Academic Группа QQ разработчиков: `610599535`
|
|
||||||
|
|
||||||
- Известные проблемы
|
|
||||||
- Некоторые расширения для браузера могут мешать работе пользовательского интерфейса этого программного обеспечения
|
|
||||||
- У официального Gradio есть много проблем совместимости, поэтому обязательно установите Gradio с помощью `requirement.txt`
|
|
||||||
|
|
||||||
### III: Темы
|
|
||||||
Вы можете изменить тему путем изменения опции `THEME` (config.py)
|
|
||||||
1. `Chuanhu-Small-and-Beautiful` [ссылка](https://github.com/GaiZhenbiao/ChuanhuChatGPT/)
|
|
||||||
|
|
||||||
|
|
||||||
### IV: Ветви разработки этого проекта
|
|
||||||
|
|
||||||
1. Ветка `master`: Основная ветка, стабильная версия
|
|
||||||
2. Ветвь `frontier`: Ветвь разработки, версия для тестирования
|
|
||||||
|
|
||||||
|
|
||||||
### V: Справочники и обучение
|
|
||||||
|
|
||||||
```
|
|
||||||
В коде использовались многие функции, представленные в других отличных проектах, поэтому их порядок не имеет значения:
|
|
||||||
|
|
||||||
# ChatGLM2-6B от Тиньхуа:
|
|
||||||
https://github.com/THUDM/ChatGLM2-6B
|
|
||||||
|
|
||||||
# Линейные модели с ограниченной памятью от Тиньхуа:
|
|
||||||
https://github.com/Jittor/JittorLLMs
|
|
||||||
|
|
||||||
# ChatPaper:
|
|
||||||
https://github.com/kaixindelele/ChatPaper
|
|
||||||
|
|
||||||
# Edge-GPT:
|
|
||||||
https://github.com/acheong08/EdgeGPT
|
|
||||||
|
|
||||||
# ChuanhuChatGPT:
|
|
||||||
https://github.com/GaiZhenbiao/ChuanhuChatGPT
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
# Установщик с одним щелчком Oobabooga:
|
|
||||||
https://github.com/oobabooga/one-click-installers
|
|
||||||
|
|
||||||
# Больше:
|
|
||||||
https://github.com/gradio-app/gradio
|
|
||||||
https://github.com/fghrsh/live2d_demo
|
|
||||||
@@ -1,26 +0,0 @@
|
|||||||
@echo off
|
|
||||||
setlocal
|
|
||||||
|
|
||||||
:: 设置环境变量
|
|
||||||
set ENV_NAME=gpt
|
|
||||||
set ENV_PATH=%~dp0%ENV_NAME%
|
|
||||||
set SCRIPT_PATH=%~dp0main.py
|
|
||||||
|
|
||||||
:: 判断环境是否已解压
|
|
||||||
if not exist "%ENV_PATH%" (
|
|
||||||
echo Extracting environment...
|
|
||||||
mkdir "%ENV_PATH%"
|
|
||||||
tar -xzf gpt.tar.gz -C "%ENV_PATH%"
|
|
||||||
|
|
||||||
:: 运行conda环境激活脚本
|
|
||||||
call "%ENV_PATH%\Scripts\activate.bat"
|
|
||||||
) else (
|
|
||||||
:: 如果环境已存在,直接激活
|
|
||||||
call "%ENV_PATH%\Scripts\activate.bat"
|
|
||||||
)
|
|
||||||
echo Start to run program:
|
|
||||||
:: 运行Python脚本
|
|
||||||
python "%SCRIPT_PATH%"
|
|
||||||
|
|
||||||
endlocal
|
|
||||||
pause
|
|
||||||
@@ -1,42 +0,0 @@
|
|||||||
# Running with fastapi
|
|
||||||
|
|
||||||
We currently support fastapi in order to solve sub-path deploy issue.
|
|
||||||
|
|
||||||
1. change CUSTOM_PATH setting in `config.py`
|
|
||||||
|
|
||||||
```sh
|
|
||||||
nano config.py
|
|
||||||
```
|
|
||||||
|
|
||||||
2. Edit main.py
|
|
||||||
|
|
||||||
```diff
|
|
||||||
auto_opentab_delay()
|
|
||||||
- demo.queue(concurrency_count=CONCURRENT_COUNT).launch(server_name="0.0.0.0", server_port=PORT, auth=AUTHENTICATION, favicon_path="docs/logo.png")
|
|
||||||
+ demo.queue(concurrency_count=CONCURRENT_COUNT)
|
|
||||||
|
|
||||||
- # 如果需要在二级路径下运行
|
|
||||||
- # CUSTOM_PATH = get_conf('CUSTOM_PATH')
|
|
||||||
- # if CUSTOM_PATH != "/":
|
|
||||||
- # from toolbox import run_gradio_in_subpath
|
|
||||||
- # run_gradio_in_subpath(demo, auth=AUTHENTICATION, port=PORT, custom_path=CUSTOM_PATH)
|
|
||||||
- # else:
|
|
||||||
- # demo.launch(server_name="0.0.0.0", server_port=PORT, auth=AUTHENTICATION, favicon_path="docs/logo.png")
|
|
||||||
|
|
||||||
+ 如果需要在二级路径下运行
|
|
||||||
+ CUSTOM_PATH = get_conf('CUSTOM_PATH')
|
|
||||||
+ if CUSTOM_PATH != "/":
|
|
||||||
+ from toolbox import run_gradio_in_subpath
|
|
||||||
+ run_gradio_in_subpath(demo, auth=AUTHENTICATION, port=PORT, custom_path=CUSTOM_PATH)
|
|
||||||
+ else:
|
|
||||||
+ demo.launch(server_name="0.0.0.0", server_port=PORT, auth=AUTHENTICATION, favicon_path="docs/logo.png")
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
```
|
|
||||||
|
|
||||||
3. Go!
|
|
||||||
|
|
||||||
```sh
|
|
||||||
python main.py
|
|
||||||
```
|
|
||||||
BIN
docs/demo.jpg
BIN
docs/demo.jpg
Binary file not shown.
|
Before Width: | Height: | Size: 262 KiB |
BIN
docs/demo2.jpg
BIN
docs/demo2.jpg
Binary file not shown.
|
Before Width: | Height: | Size: 264 KiB |
@@ -1,189 +0,0 @@
|
|||||||
# 实现带二级菜单的插件
|
|
||||||
|
|
||||||
## 一、如何写带有二级菜单的插件
|
|
||||||
|
|
||||||
1. 声明一个 `Class`，继承父类 `GptAcademicPluginTemplate`
|
|
||||||
|
|
||||||
```python
|
|
||||||
from crazy_functions.plugin_template.plugin_class_template import GptAcademicPluginTemplate
|
|
||||||
from crazy_functions.plugin_template.plugin_class_template import ArgProperty
|
|
||||||
|
|
||||||
class Demo_Wrap(GptAcademicPluginTemplate):
|
|
||||||
def __init__(self): ...
|
|
||||||
```
|
|
||||||
|
|
||||||
2. 声明二级菜单中需要的变量，覆盖父类的`define_arg_selection_menu`函数。
|
|
||||||
|
|
||||||
```python
|
|
||||||
class Demo_Wrap(GptAcademicPluginTemplate):
|
|
||||||
...
|
|
||||||
|
|
||||||
def define_arg_selection_menu(self):
|
|
||||||
"""
|
|
||||||
定义插件的二级选项菜单
|
|
||||||
|
|
||||||
第一个参数，名称`main_input`，参数`type`声明这是一个文本框，文本框上方显示`title`，文本框内部显示`description`，`default_value`为默认值；
|
|
||||||
第二个参数，名称`advanced_arg`，参数`type`声明这是一个文本框，文本框上方显示`title`，文本框内部显示`description`，`default_value`为默认值；
|
|
||||||
第三个参数，名称`allow_cache`，参数`type`声明这是一个下拉菜单，下拉菜单上方显示`title`+`description`，下拉菜单的选项为`options`，`default_value`为下拉菜单默认值；
|
|
||||||
"""
|
|
||||||
gui_definition = {
|
|
||||||
"main_input":
|
|
||||||
ArgProperty(title="ArxivID", description="输入Arxiv的ID或者网址", default_value="", type="string").model_dump_json(),
|
|
||||||
"advanced_arg":
|
|
||||||
ArgProperty(title="额外的翻译提示词",
|
|
||||||
description=r"如果有必要, 请在此处给出自定义翻译命令",
|
|
||||||
default_value="", type="string").model_dump_json(),
|
|
||||||
"allow_cache":
|
|
||||||
ArgProperty(title="是否允许从缓存中调取结果", options=["允许缓存", "从头执行"], default_value="允许缓存", description="无", type="dropdown").model_dump_json(),
|
|
||||||
}
|
|
||||||
return gui_definition
|
|
||||||
...
|
|
||||||
```
|
|
||||||
|
|
||||||
|
|
||||||
> [!IMPORTANT]
|
|
||||||
>
|
|
||||||
> ArgProperty 中每个条目对应一个参数，`type == "string"`时，使用文本块，`type == dropdown`时，使用下拉菜单。
|
|
||||||
>
|
|
||||||
> 注意：`main_input` 和 `advanced_arg`是两个特殊的参数。`main_input`会自动与界面右上角的`输入区`进行同步，而`advanced_arg`会自动与界面右下角的`高级参数输入区`同步。除此之外，参数名称可以任意选取。其他细节详见`crazy_functions/plugin_template/plugin_class_template.py`。
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
3. 编写插件程序，覆盖父类的`execute`函数。
|
|
||||||
|
|
||||||
例如：
|
|
||||||
|
|
||||||
```python
|
|
||||||
class Demo_Wrap(GptAcademicPluginTemplate):
|
|
||||||
...
|
|
||||||
...
|
|
||||||
|
|
||||||
def execute(txt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_request):
|
|
||||||
"""
|
|
||||||
执行插件
|
|
||||||
|
|
||||||
plugin_kwargs字典中会包含用户的选择，与上述 `define_arg_selection_menu` 一一对应
|
|
||||||
"""
|
|
||||||
allow_cache = plugin_kwargs["allow_cache"]
|
|
||||||
advanced_arg = plugin_kwargs["advanced_arg"]
|
|
||||||
|
|
||||||
if allow_cache == "从头执行": plugin_kwargs["advanced_arg"] = "--no-cache " + plugin_kwargs["advanced_arg"]
|
|
||||||
yield from Latex翻译中文并重新编译PDF(txt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_request)
|
|
||||||
|
|
||||||
```
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
4. 注册插件
|
|
||||||
|
|
||||||
将以下条目插入`crazy_functional.py`即可。注意，与旧插件不同的是，`Function`键值应该为None，而`Class`键值为上述插件的类名称（`Demo_Wrap`）。
|
|
||||||
```
|
|
||||||
"新插件": {
|
|
||||||
"Group": "学术",
|
|
||||||
"Color": "stop",
|
|
||||||
"AsButton": True,
|
|
||||||
"Info": "插件说明",
|
|
||||||
"Function": None,
|
|
||||||
"Class": Demo_Wrap,
|
|
||||||
},
|
|
||||||
```
|
|
||||||
|
|
||||||
5. 已经结束了，启动程序测试吧~！
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
## 二、背后的原理（需要JavaScript的前置知识）
|
|
||||||
|
|
||||||
|
|
||||||
### (I) 首先介绍三个Gradio官方没有的重要前端函数
|
|
||||||
|
|
||||||
主javascript程序`common.js`中有三个Gradio官方没有的重要API
|
|
||||||
|
|
||||||
1. `get_data_from_gradio_component`
|
|
||||||
这个函数可以获取任意gradio组件的当前值，例如textbox中的字符，dropdown中的当前选项，chatbot当前的对话等等。调用方法举例：
|
|
||||||
```javascript
|
|
||||||
// 获取当前的对话
|
|
||||||
let chatbot = await get_data_from_gradio_component('gpt-chatbot');
|
|
||||||
```
|
|
||||||
|
|
||||||
2. `get_gradio_component`
|
|
||||||
有时候我们不仅需要gradio组件的当前值，还需要它的label值、是否隐藏、下拉菜单其他可选选项等等，而通过这个函数可以直接获取这个组件的句柄。举例：
|
|
||||||
```javascript
|
|
||||||
// 获取下拉菜单组件的句柄
|
|
||||||
var model_sel = await get_gradio_component("elem_model_sel");
|
|
||||||
// 获取它的所有属性，包括其所有可选选项
|
|
||||||
console.log(model_sel.props)
|
|
||||||
```
|
|
||||||
|
|
||||||
|
|
||||||
3. `push_data_to_gradio_component`
|
|
||||||
这个函数可以将数据推回gradio组件，例如textbox中的字符，dropdown中的当前选项等等。调用方法举例：
|
|
||||||
|
|
||||||
```javascript
|
|
||||||
// 修改一个按钮上面的文本
|
|
||||||
push_data_to_gradio_component("btnName", "gradio_element_id", "string");
|
|
||||||
|
|
||||||
// 隐藏一个组件
|
|
||||||
push_data_to_gradio_component({ visible: false, __type__: 'update' }, "plugin_arg_menu", "obj");
|
|
||||||
|
|
||||||
// 修改组件label
|
|
||||||
push_data_to_gradio_component({ label: '新label的值', __type__: 'update' }, "gpt-chatbot", "obj")
|
|
||||||
|
|
||||||
// 第一个参数是value，
|
|
||||||
// - 可以是字符串（调整textbox的文本，按钮的文本）；
|
|
||||||
// - 还可以是 { visible: false, __type__: 'update' } 这样的字典（调整visible, label, choices）
|
|
||||||
// 第二个参数是elem_id
|
|
||||||
// 第三个参数是"string" 或者 "obj"
|
|
||||||
```
|
|
||||||
|
|
||||||
|
|
||||||
### (II) 从点击插件到执行插件的逻辑过程
|
|
||||||
|
|
||||||
简述：程序启动时把每个插件的二级菜单编码为BASE64，存储在用户的浏览器前端，用户调用对应功能时，会按照插件的BASE64编码，将平时隐藏的菜单（有选择性地）显示出来。
|
|
||||||
|
|
||||||
1. 启动阶段（主函数 `main.py` 中），遍历每个插件，生成二级菜单的BASE64编码，存入变量`register_advanced_plugin_init_code_arr`。
|
|
||||||
```python
|
|
||||||
def get_js_code_for_generating_menu(self, btnName):
|
|
||||||
define_arg_selection = self.define_arg_selection_menu()
|
|
||||||
DEFINE_ARG_INPUT_INTERFACE = json.dumps(define_arg_selection)
|
|
||||||
return base64.b64encode(DEFINE_ARG_INPUT_INTERFACE.encode('utf-8')).decode('utf-8')
|
|
||||||
```
|
|
||||||
|
|
||||||
|
|
||||||
2. 用户加载阶段（主javascript程序`common.js`中），浏览器加载`register_advanced_plugin_init_code_arr`，存入本地的字典`advanced_plugin_init_code_lib`：
|
|
||||||
|
|
||||||
```javascript
|
|
||||||
advanced_plugin_init_code_lib = {}
|
|
||||||
function register_advanced_plugin_init_code(key, code){
|
|
||||||
advanced_plugin_init_code_lib[key] = code;
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
3. 用户点击插件按钮（主函数 `main.py` 中）时，仅执行以下javascript代码，唤醒隐藏的二级菜单（生成菜单的代码在`common.js`中的`generate_menu`函数上）：
|
|
||||||
|
|
||||||
|
|
||||||
```javascript
|
|
||||||
// 生成高级插件的选择菜单
|
|
||||||
function run_advanced_plugin_launch_code(key){
|
|
||||||
generate_menu(advanced_plugin_init_code_lib[key], key);
|
|
||||||
}
|
|
||||||
function on_flex_button_click(key){
|
|
||||||
run_advanced_plugin_launch_code(key);
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
```python
|
|
||||||
click_handle = plugins[k]["Button"].click(None, inputs=[], outputs=None, _js=f"""()=>run_advanced_plugin_launch_code("{k}")""")
|
|
||||||
```
|
|
||||||
|
|
||||||
4. 当用户点击二级菜单的执行键时，通过javascript脚本模拟点击一个隐藏按钮，触发后续程序（`common.js`中的`execute_current_pop_up_plugin`，会把二级菜单中的参数缓存到`invisible_current_pop_up_plugin_arg_final`，然后模拟点击`invisible_callback_btn_for_plugin_exe`按钮）。隐藏按钮的定义在（主函数 `main.py` ），该隐藏按钮会最终触发`route_switchy_bt_with_arg`函数（定义于`themes/gui_advanced_plugin_class.py`）：
|
|
||||||
|
|
||||||
```python
|
|
||||||
click_handle_ng = new_plugin_callback.click(route_switchy_bt_with_arg, [
|
|
||||||
gr.State(["new_plugin_callback", "usr_confirmed_arg"] + input_combo_order),
|
|
||||||
new_plugin_callback, usr_confirmed_arg, *input_combo
|
|
||||||
], output_combo)
|
|
||||||
```
|
|
||||||
|
|
||||||
5. 最后，`route_switchy_bt_with_arg`中，会搜集所有用户参数，统一集中到`plugin_kwargs`参数中，并执行对应插件的`execute`函数。
|
|
||||||
@@ -1,378 +0,0 @@
|
|||||||
# chatgpt-academic项目自译解报告
|
|
||||||
（Author补充：以下分析均由本项目调用ChatGPT一键生成，如果有不准确的地方，全怪GPT😄）
|
|
||||||
|
|
||||||
|
|
||||||
| 文件名 | 功能描述 |
|
|
||||||
| ------ | ------ |
|
|
||||||
| check_proxy.py | 检查代理有效性及地理位置 |
|
|
||||||
| colorful.py | 控制台打印彩色文字 |
|
|
||||||
| config.py | 配置和参数设置 |
|
|
||||||
| config_private.py | 私人配置和参数设置 |
|
|
||||||
| core_functional.py | 核心函数和参数设置 |
|
|
||||||
| crazy_functional.py | 高级功能插件集合 |
|
|
||||||
| main.py | 一个 Chatbot 程序，提供各种学术翻译、文本处理和其他查询服务 |
|
|
||||||
| multi_language.py | 识别和翻译不同语言 |
|
|
||||||
| theme.py | 自定义 gradio 应用程序主题 |
|
|
||||||
| toolbox.py | 工具类库，用于协助实现各种功能 |
|
|
||||||
| crazy_functions\crazy_functions_test.py | 测试 crazy_functions 中的各种函数 |
|
|
||||||
| crazy_functions\crazy_utils.py | 工具函数，用于字符串处理、异常检测、Markdown 格式转换等 |
|
|
||||||
| crazy_functions\Latex全文润色.py | 对整个 Latex 项目进行润色和纠错 |
|
|
||||||
| crazy_functions\Latex全文翻译.py | 对整个 Latex 项目进行翻译 |
|
|
||||||
| crazy_functions\\_\_init\_\_.py | 模块初始化文件，标识 `crazy_functions` 是一个包 |
|
|
||||||
| crazy_functions\下载arxiv论文翻译摘要.py | 下载 `arxiv` 论文的 PDF 文件，并提取摘要和翻译 |
|
|
||||||
| crazy_functions\代码重写为全英文_多线程.py | 将Python源代码文件中的中文内容转化为英文 |
|
|
||||||
| crazy_functions\图片生成.py | 根据激励文本使用GPT模型生成相应的图像 |
|
|
||||||
| crazy_functions\Conversation_To_File.py | 将每次对话记录写入Markdown格式的文件中 |
|
|
||||||
| crazy_functions\总结word文档.py | 对输入的word文档进行摘要生成 |
|
|
||||||
| crazy_functions\总结音视频.py | 对输入的音视频文件进行摘要生成 |
|
|
||||||
| crazy_functions\Markdown_Translate.py | 将指定目录下的Markdown文件进行中英文翻译 |
|
|
||||||
| crazy_functions\批量总结PDF文档.py | 对PDF文件进行切割和摘要生成 |
|
|
||||||
| crazy_functions\批量总结PDF文档pdfminer.py | 对PDF文件进行文本内容的提取和摘要生成 |
|
|
||||||
| crazy_functions\PDF_Translate.py | 将指定目录下的PDF文件进行中英文翻译 |
|
|
||||||
| crazy_functions\理解PDF文档内容.py | 对PDF文件进行摘要生成和问题解答 |
|
|
||||||
| crazy_functions\生成函数注释.py | 自动生成Python函数的注释 |
|
|
||||||
| crazy_functions\联网的ChatGPT.py | 使用网络爬虫和ChatGPT模型进行聊天回答 |
|
|
||||||
| crazy_functions\解析JupyterNotebook.py | 对Jupyter Notebook进行代码解析 |
|
|
||||||
| crazy_functions\解析项目源代码.py | 对指定编程语言的源代码进行解析 |
|
|
||||||
| crazy_functions\询问多个大语言模型.py | 使用多个大语言模型对输入进行处理和回复 |
|
|
||||||
| crazy_functions\读文章写摘要.py | 对论文进行解析和全文摘要生成 |
|
|
||||||
| crazy_functions\谷歌检索小助手.py | 提供谷歌学术搜索页面中相关文章的元数据信息。 |
|
|
||||||
| crazy_functions\高级功能函数模板.py | 使用Unsplash API发送相关图片以回复用户的输入。 |
|
|
||||||
| request_llms\bridge_all.py | 基于不同LLM模型进行对话。 |
|
|
||||||
| request_llms\bridge_chatglm.py | 使用ChatGLM模型生成回复，支持单线程和多线程方式。 |
|
|
||||||
| request_llms\bridge_chatgpt.py | 基于GPT模型完成对话。 |
|
|
||||||
| request_llms\bridge_jittorllms_llama.py | 使用JittorLLMs模型完成对话，支持单线程和多线程方式。 |
|
|
||||||
| request_llms\bridge_jittorllms_pangualpha.py | 使用JittorLLMs模型完成对话，基于多进程和多线程方式。 |
|
|
||||||
| request_llms\bridge_jittorllms_rwkv.py | 使用JittorLLMs模型完成聊天功能，提供包括历史信息、参数调节等在内的多个功能选项。 |
|
|
||||||
| request_llms\bridge_moss.py | 加载Moss模型完成对话功能。 |
|
|
||||||
| request_llms\bridge_newbing.py | 使用Newbing聊天机器人进行对话，支持单线程和多线程方式。 |
|
|
||||||
| request_llms\bridge_newbingfree.py | 基于Bing chatbot API实现聊天机器人的文本生成功能。 |
|
|
||||||
| request_llms\bridge_stackclaude.py | 基于Slack API实现Claude与用户的交互。 |
|
|
||||||
| request_llms\bridge_tgui.py | 通过websocket实现聊天机器人与UI界面交互。 |
|
|
||||||
| request_llms\edge_gpt.py | 调用Bing chatbot API提供聊天机器人服务。 |
|
|
||||||
| request_llms\edge_gpt_free.py | 实现聊天机器人API，采用aiohttp和httpx工具库。 |
|
|
||||||
| request_llms\test_llms.py | 对llm模型进行单元测试。 |
|
|
||||||
|
|
||||||
## 接下来请你逐文件分析下面的工程[0/48] 请对下面的程序文件做一个概述: check_proxy.py
|
|
||||||
|
|
||||||
这个文件主要包含了五个函数：
|
|
||||||
|
|
||||||
1. `check_proxy`：用于检查代理的有效性及地理位置，输出代理配置和所在地信息。
|
|
||||||
|
|
||||||
2. `backup_and_download`：用于备份当前版本并下载新版本。
|
|
||||||
|
|
||||||
3. `patch_and_restart`：用于覆盖更新当前版本并重新启动程序。
|
|
||||||
|
|
||||||
4. `get_current_version`：用于获取当前程序的版本号。
|
|
||||||
|
|
||||||
5. `auto_update`：用于自动检查新版本并提示用户更新。如果用户选择更新，则备份并下载新版本，覆盖更新当前版本并重新启动程序。如果更新失败，则输出错误信息，并不会向用户进行任何提示。
|
|
||||||
|
|
||||||
还有一个没有函数名的语句`os.environ['no_proxy'] = '*'`，用于设置环境变量，避免代理网络产生意外污染。
|
|
||||||
|
|
||||||
此外，该文件导入了以下三个模块/函数：
|
|
||||||
|
|
||||||
- `requests`
|
|
||||||
- `shutil`
|
|
||||||
- `os`
|
|
||||||
|
|
||||||
## [1/48] 请对下面的程序文件做一个概述: colorful.py
|
|
||||||
|
|
||||||
该文件是一个Python脚本，用于在控制台中打印彩色文字。该文件包含了一些函数，用于以不同颜色打印文本。其中，红色、绿色、黄色、蓝色、紫色、靛色分别以函数 print红、print绿、print黄、print蓝、print紫、print靛 的形式定义；亮红色、亮绿色、亮黄色、亮蓝色、亮紫色、亮靛色分别以 print亮红、print亮绿、print亮黄、print亮蓝、print亮紫、print亮靛 的形式定义。它们使用 ANSI Escape Code 将彩色输出从控制台突出显示。如果运行在 Linux 操作系统上，文件所执行的操作被留空；否则，该文件导入了 colorama 库并调用 init() 函数进行初始化。最后，通过一系列条件语句，该文件通过将所有彩色输出函数的名称重新赋值为 print 函数的名称来避免输出文件的颜色问题。
|
|
||||||
|
|
||||||
## [2/48] 请对下面的程序文件做一个概述: config.py
|
|
||||||
|
|
||||||
这个程序文件是用来配置和参数设置的。它包含了许多设置，如API key，使用代理，线程数，默认模型，超时时间等等。此外，它还包含了一些高级功能，如URL重定向等。这些设置将会影响到程序的行为和性能。
|
|
||||||
|
|
||||||
## [3/48] 请对下面的程序文件做一个概述: config_private.py
|
|
||||||
|
|
||||||
这个程序文件是一个Python脚本，文件名为config_private.py。其中包含以下变量的赋值：
|
|
||||||
|
|
||||||
1. API_KEY：API密钥。
|
|
||||||
2. USE_PROXY：是否应用代理。
|
|
||||||
3. proxies：如果使用代理，则设置代理网络的协议(socks5/http)、地址(localhost)和端口(11284)。
|
|
||||||
4. DEFAULT_WORKER_NUM：默认的工作线程数量。
|
|
||||||
5. SLACK_CLAUDE_BOT_ID：Slack机器人ID。
|
|
||||||
6. SLACK_CLAUDE_USER_TOKEN：Slack用户令牌。
|
|
||||||
|
|
||||||
## [4/48] 请对下面的程序文件做一个概述: core_functional.py
|
|
||||||
|
|
||||||
这是一个名为core_functional.py的源代码文件，该文件定义了一个名为get_core_functions()的函数，该函数返回一个字典，该字典包含了各种学术翻译润色任务的说明和相关参数，如颜色、前缀、后缀等。这些任务包括英语学术润色、中文学术润色、查找语法错误、中译英、学术中英互译、英译中、找图片和参考文献转Bib。其中，一些任务还定义了预处理函数用于处理任务的输入文本。
|
|
||||||
|
|
||||||
## [5/48] 请对下面的程序文件做一个概述: crazy_functional.py
|
|
||||||
|
|
||||||
此程序文件（crazy_functional.py）是一个函数插件集合，包含了多个函数插件的定义和调用。这些函数插件旨在提供一些高级功能，如解析项目源代码、批量翻译PDF文档和Latex全文润色等。其中一些插件还支持热更新功能，不需要重启程序即可生效。文件中的函数插件按照功能进行了分类（第一组和第二组），并且有不同的调用方式（作为按钮或下拉菜单）。
|
|
||||||
|
|
||||||
## [6/48] 请对下面的程序文件做一个概述: main.py
|
|
||||||
|
|
||||||
这是一个Python程序文件，文件名为main.py。该程序包含一个名为main的函数，程序会自动运行该函数。程序要求已经安装了gradio、os等模块，会根据配置文件加载代理、model、API Key等信息。程序提供了Chatbot功能，实现了一个对话界面，用户可以输入问题，然后Chatbot可以回答问题或者提供相关功能。程序还包含了基础功能区、函数插件区、更换模型 & SysPrompt & 交互界面布局、备选输入区，用户可以在这些区域选择功能和插件进行使用。程序中还包含了一些辅助模块，如logging等。
|
|
||||||
|
|
||||||
## [7/48] 请对下面的程序文件做一个概述: multi_language.py
|
|
||||||
|
|
||||||
该文件multi_language.py是用于将项目翻译成不同语言的程序。它包含了以下函数和变量：lru_file_cache、contains_chinese、split_list、map_to_json、read_map_from_json、advanced_split、trans、trans_json、step_1_core_key_translate、CACHE_FOLDER、blacklist、LANG、TransPrompt、cached_translation等。注释和文档字符串提供了有关程序的说明，例如如何使用该程序，如何修改“LANG”和“TransPrompt”变量等。
|
|
||||||
|
|
||||||
## [8/48] 请对下面的程序文件做一个概述: theme.py
|
|
||||||
|
|
||||||
这是一个Python源代码文件，文件名为theme.py。此文件中定义了一个函数adjust_theme，其功能是自定义gradio应用程序的主题，包括调整颜色、字体、阴影等。如果允许，则添加一个看板娘。此文件还包括变量advanced_css，其中包含一些CSS样式，用于高亮显示代码和自定义聊天框样式。此文件还导入了get_conf函数和gradio库。
|
|
||||||
|
|
||||||
## [9/48] 请对下面的程序文件做一个概述: toolbox.py
|
|
||||||
|
|
||||||
toolbox.py是一个工具类库，其中主要包含了一些函数装饰器和小工具函数，用于协助实现聊天机器人所需的各种功能，包括文本处理、功能插件加载、异常检测、Markdown格式转换，文件读写等等。此外，该库还包含一些依赖、参数配置等信息。该库易于理解和维护。
|
|
||||||
|
|
||||||
## [10/48] 请对下面的程序文件做一个概述: crazy_functions\crazy_functions_test.py
|
|
||||||
|
|
||||||
这个文件是一个Python测试模块，用于测试crazy_functions中的各种函数插件。这些函数包括：解析Python项目源代码、解析Cpp项目源代码、Latex全文润色、Markdown中译英、批量翻译PDF文档、谷歌检索小助手、总结word文档、下载arxiv论文并翻译摘要、联网回答问题、和解析Jupyter Notebooks。对于每个函数插件，都有一个对应的测试函数来进行测试。
|
|
||||||
|
|
||||||
## [11/48] 请对下面的程序文件做一个概述: crazy_functions\crazy_utils.py
|
|
||||||
|
|
||||||
这个Python文件中包括了两个函数：
|
|
||||||
|
|
||||||
1. `input_clipping`: 该函数用于裁剪输入文本长度，使其不超过一定的限制。
|
|
||||||
2. `request_gpt_model_in_new_thread_with_ui_alive`: 该函数用于请求 GPT 模型并保持用户界面的响应，支持多线程和实时更新用户界面。
|
|
||||||
|
|
||||||
这两个函数都依赖于从 `toolbox` 和 `request_llms` 中导入的一些工具函数。函数的输入和输出有详细的描述文档。
|
|
||||||
|
|
||||||
## [12/48] 请对下面的程序文件做一个概述: crazy_functions\Latex全文润色.py
|
|
||||||
|
|
||||||
这是一个Python程序文件，文件名为crazy_functions\Latex全文润色.py。文件包含了一个PaperFileGroup类和三个函数Latex英文润色，Latex中文润色和Latex英文纠错。程序使用了字符串处理、正则表达式、文件读写、多线程等技术，主要作用是对整个Latex项目进行润色和纠错。其中润色和纠错涉及到了对文本的语法、清晰度和整体可读性等方面的提升。此外，该程序还参考了第三方库，并封装了一些工具函数。
|
|
||||||
|
|
||||||
## [13/48] 请对下面的程序文件做一个概述: crazy_functions\Latex全文翻译.py
|
|
||||||
|
|
||||||
这个文件包含两个函数 `Latex英译中` 和 `Latex中译英`，它们都会对整个Latex项目进行翻译。这个文件还包含一个类 `PaperFileGroup`，它拥有一个方法 `run_file_split`，用于把长文本文件分成多个短文件。其中使用了工具库 `toolbox` 中的一些函数和从 `request_llms` 中导入了 `model_info`。接下来的函数把文件读取进来，把它们的注释删除，进行分割，并进行翻译。这个文件还包括了一些异常处理和界面更新的操作。
|
|
||||||
|
|
||||||
## [14/48] 请对下面的程序文件做一个概述: crazy_functions\__init__.py
|
|
||||||
|
|
||||||
这是一个Python模块的初始化文件（__init__.py），命名为"crazy_functions"。该模块包含了一些疯狂的函数，但该文件并没有实现这些函数，而是作为一个包（package）来导入其它的Python模块以实现这些函数。在该文件中，没有定义任何类或函数，它唯一的作用就是标识"crazy_functions"模块是一个包。
|
|
||||||
|
|
||||||
## [15/48] 请对下面的程序文件做一个概述: crazy_functions\下载arxiv论文翻译摘要.py
|
|
||||||
|
|
||||||
这是一个 Python 程序文件，文件名为 `下载arxiv论文翻译摘要.py`。程序包含多个函数，其中 `下载arxiv论文并翻译摘要` 函数的作用是下载 `arxiv` 论文的 PDF 文件，提取摘要并使用 GPT 对其进行翻译。其他函数包括用于下载 `arxiv` 论文的 `download_arxiv_` 函数和用于获取文章信息的 `get_name` 函数，其中涉及使用第三方库如 requests, BeautifulSoup 等。该文件还包含一些用于调试和存储文件的代码段。
|
|
||||||
|
|
||||||
## [16/48] 请对下面的程序文件做一个概述: crazy_functions\代码重写为全英文_多线程.py
|
|
||||||
|
|
||||||
该程序文件是一个多线程程序，主要功能是将指定目录下的所有Python代码文件中的中文内容转化为英文，并将转化后的代码存储到一个新的文件中。其中，程序使用了GPT-3等技术进行中文-英文的转化，同时也进行了一些Token限制下的处理，以防止程序发生错误。程序在执行过程中还会输出一些提示信息，并将所有转化过的代码文件存储到指定目录下。在程序执行结束后，还会生成一个任务执行报告，记录程序运行的详细信息。
|
|
||||||
|
|
||||||
## [17/48] 请对下面的程序文件做一个概述: crazy_functions\图片生成.py
|
|
||||||
|
|
||||||
该程序文件提供了一个用于生成图像的函数`图片生成`。函数实现的过程中，会调用`gen_image`函数来生成图像，并返回图像生成的网址和本地文件地址。函数有多个参数，包括`prompt`(激励文本)、`llm_kwargs`(GPT模型的参数)、`plugin_kwargs`(插件模型的参数)等。函数核心代码使用了`requests`库向OpenAI API请求图像，并做了简单的处理和保存。函数还更新了交互界面，清空聊天历史并显示正在生成图像的消息和最终的图像网址和预览。
|
|
||||||
|
|
||||||
## [18/48] 请对下面的程序文件做一个概述: crazy_functions\Conversation_To_File.py
|
|
||||||
|
|
||||||
这个文件是名为crazy_functions\Conversation_To_File.py的Python程序文件，包含了4个函数：
|
|
||||||
|
|
||||||
1. write_chat_to_file(chatbot, history=None, file_name=None)：用来将对话记录以Markdown格式写入文件中，并且生成文件名，如果没指定文件名则用当前时间。写入完成后将文件路径打印出来。
|
|
||||||
|
|
||||||
2. gen_file_preview(file_name)：从传入的文件中读取内容，解析出对话历史记录并返回前100个字符，用于文件预览。
|
|
||||||
|
|
||||||
3. read_file_to_chat(chatbot, history, file_name)：从传入的文件中读取内容，解析出对话历史记录并更新聊天显示框。
|
|
||||||
|
|
||||||
4. Conversation_To_File(txt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_request)：一个主要函数，用于保存当前对话记录并提醒用户。如果用户希望加载历史记录，则调用read_file_to_chat()来更新聊天显示框。如果用户希望删除历史记录，调用删除所有本地对话历史记录()函数完成删除操作。
|
|
||||||
|
|
||||||
## [19/48] 请对下面的程序文件做一个概述: crazy_functions\总结word文档.py
|
|
||||||
|
|
||||||
该程序文件实现了一个总结Word文档的功能，使用Python的docx库读取docx格式的文件，使用pywin32库读取doc格式的文件。程序会先根据传入的txt参数搜索需要处理的文件，并逐个解析其中的内容，将内容拆分为指定长度的文章片段，然后使用另一个程序文件中的request_gpt_model_in_new_thread_with_ui_alive函数进行中文概述。最后将所有的总结结果写入一个文件中，并在界面上进行展示。
|
|
||||||
|
|
||||||
## [20/48] 请对下面的程序文件做一个概述: crazy_functions\总结音视频.py
|
|
||||||
|
|
||||||
该程序文件包括两个函数：split_audio_file()和AnalyAudio()，并且导入了一些必要的库并定义了一些工具函数。split_audio_file用于将音频文件分割成多个时长相等的片段，返回一个包含所有切割音频片段文件路径的列表，而AnalyAudio用来分析音频文件，通过调用whisper模型进行音频转文字并使用GPT模型对音频内容进行概述，最终将所有总结结果写入结果文件中。
|
|
||||||
|
|
||||||
## [21/48] 请对下面的程序文件做一个概述: crazy_functions\Markdown_Translate.py
|
|
||||||
|
|
||||||
该程序文件名为`Markdown_Translate.py`，包含了以下功能：读取Markdown文件，将长文本分离开来，将Markdown文件进行翻译（英译中和中译英），整理结果并退出。程序使用了多线程以提高效率。程序使用了`tiktoken`依赖库，可能需要额外安装。文件中还有一些其他的函数和类，但与文件名所描述的功能无关。
|
|
||||||
|
|
||||||
## [22/48] 请对下面的程序文件做一个概述: crazy_functions\批量总结PDF文档.py
|
|
||||||
|
|
||||||
该文件是一个Python脚本，名为crazy_functions\批量总结PDF文档.py。在导入了一系列库和工具函数后，主要定义了5个函数，其中包括一个错误处理装饰器（@CatchException），用于批量总结PDF文档。该函数主要实现对PDF文档的解析，并调用模型生成中英文摘要。
|
|
||||||
|
|
||||||
## [23/48] 请对下面的程序文件做一个概述: crazy_functions\批量总结PDF文档pdfminer.py
|
|
||||||
|
|
||||||
该程序文件是一个用于批量总结PDF文档的函数插件，使用了pdfminer插件和BeautifulSoup库来提取PDF文档的文本内容，对每个PDF文件分别进行处理并生成中英文摘要。同时，该程序文件还包括一些辅助工具函数和处理异常的装饰器。
|
|
||||||
|
|
||||||
## [24/48] 请对下面的程序文件做一个概述: crazy_functions\PDF_Translate.py
|
|
||||||
|
|
||||||
这个程序文件是一个Python脚本，文件名为“PDF_Translate.py”。它主要使用了“toolbox”、“request_gpt_model_in_new_thread_with_ui_alive”、“request_gpt_model_multi_threads_with_very_awesome_ui_and_high_efficiency”、“colorful”等Python库和自定义的模块“crazy_utils”的一些函数。程序实现了一个批量翻译PDF文档的功能，可以自动解析PDF文件中的基础信息，递归地切割PDF文件，翻译和处理PDF论文中的所有内容，并生成相应的翻译结果文件（包括md文件和html文件）。功能比较复杂，其中需要调用多个函数和依赖库，涉及到多线程操作和UI更新。文件中有详细的注释和变量命名，代码比较清晰易读。
|
|
||||||
|
|
||||||
## [25/48] 请对下面的程序文件做一个概述: crazy_functions\理解PDF文档内容.py
|
|
||||||
|
|
||||||
该程序文件实现了一个名为“理解PDF文档内容”的函数，该函数可以为输入的PDF文件提取摘要以及正文各部分的主要内容，并在提取过程中根据上下文关系进行学术性问题解答。该函数依赖于多个辅助函数和第三方库，并在执行过程中针对可能出现的异常进行了处理。
|
|
||||||
|
|
||||||
## [26/48] 请对下面的程序文件做一个概述: crazy_functions\生成函数注释.py
|
|
||||||
|
|
||||||
该程序文件是一个Python模块文件，文件名为“生成函数注释.py”，定义了两个函数：一个是生成函数注释的主函数“生成函数注释”，另一个是通过装饰器实现异常捕捉的函数“批量生成函数注释”。该程序文件依赖于“toolbox”和本地“crazy_utils”模块，并且在运行时使用了多线程技术和GPT模型来生成注释。函数生成的注释结果使用Markdown表格输出并写入历史记录文件。
|
|
||||||
|
|
||||||
## [27/48] 请对下面的程序文件做一个概述: crazy_functions\联网的ChatGPT.py
|
|
||||||
|
|
||||||
这是一个名为`联网的ChatGPT.py`的Python程序文件，其中定义了一个函数`连接网络回答问题`。该函数通过爬取搜索引擎的结果和访问网页来综合回答给定的问题，并使用ChatGPT模型完成回答。此外，该文件还包括一些工具函数，例如从网页中抓取文本和使用代理访问网页。
|
|
||||||
|
|
||||||
## [28/48] 请对下面的程序文件做一个概述: crazy_functions\解析JupyterNotebook.py
|
|
||||||
|
|
||||||
这个程序文件包含了两个函数： `parseNotebook()`和`解析ipynb文件()`，并且引入了一些工具函数和类。`parseNotebook()`函数将Jupyter Notebook文件解析为文本代码块，`解析ipynb文件()`函数则用于解析多个Jupyter Notebook文件，使用`parseNotebook()`解析每个文件和一些其他的处理。函数中使用了多线程处理输入和输出，并且将结果写入到文件中。
|
|
||||||
|
|
||||||
## [29/48] 请对下面的程序文件做一个概述: crazy_functions\解析项目源代码.py
|
|
||||||
|
|
||||||
这是一个源代码分析的Python代码文件，其中定义了多个函数，包括解析一个Python项目、解析一个C项目、解析一个C项目的头文件和解析一个Java项目等。其中解析源代码新函数是实际处理源代码分析并生成报告的函数。该函数首先会逐个读取传入的源代码文件，生成对应的请求内容，通过多线程发送到chatgpt进行分析。然后将结果写入文件，并进行汇总分析。最后通过调用update_ui函数刷新界面，完整实现了源代码的分析。
|
|
||||||
|
|
||||||
## [30/48] 请对下面的程序文件做一个概述: crazy_functions\询问多个大语言模型.py
|
|
||||||
|
|
||||||
该程序文件包含两个函数：同时问询()和同时问询_指定模型()，它们的作用是使用多个大语言模型同时对用户输入进行处理，返回对应模型的回复结果。同时问询()会默认使用ChatGPT和ChatGLM两个模型，而同时问询_指定模型()则可以指定要使用的模型。该程序文件还引用了其他的模块和函数库。
|
|
||||||
|
|
||||||
## [31/48] 请对下面的程序文件做一个概述: crazy_functions\读文章写摘要.py
|
|
||||||
|
|
||||||
这个程序文件是一个Python模块，文件名为crazy_functions\读文章写摘要.py。该模块包含了两个函数，其中主要函数是"读文章写摘要"函数，其实现了解析给定文件夹中的tex文件，对其中每个文件的内容进行摘要生成，并根据各论文片段的摘要，最终生成全文摘要。第二个函数是"解析Paper"函数，用于解析单篇论文文件。其中用到了一些工具函数和库，如update_ui、CatchException、report_exception、write_results_to_file等。
|
|
||||||
|
|
||||||
## [32/48] 请对下面的程序文件做一个概述: crazy_functions\谷歌检索小助手.py
|
|
||||||
|
|
||||||
该文件是一个Python模块，文件名为“谷歌检索小助手.py”。该模块包含两个函数，一个是“get_meta_information()”，用于从提供的网址中分析出所有相关的学术文献的元数据信息；另一个是“谷歌检索小助手()”，是主函数，用于分析用户提供的谷歌学术搜索页面中出现的文章，并提取相关信息。其中，“谷歌检索小助手()”函数依赖于“get_meta_information()”函数，并调用了其他一些Python模块，如“arxiv”、“math”、“bs4”等。
|
|
||||||
|
|
||||||
## [33/48] 请对下面的程序文件做一个概述: crazy_functions\高级功能函数模板.py
|
|
||||||
|
|
||||||
该程序文件定义了一个名为高阶功能模板函数的函数，该函数接受多个参数，包括输入的文本、gpt模型参数、插件模型参数、聊天显示框的句柄、聊天历史等，并利用送出请求，使用 Unsplash API 发送相关图片。其中，为了避免输入溢出，函数会在开始时清空历史。函数也有一些 UI 更新的语句。该程序文件还依赖于其他两个模块：CatchException 和 update_ui，以及一个名为 request_gpt_model_in_new_thread_with_ui_alive 的来自 crazy_utils 模块（应该是自定义的工具包）的函数。
|
|
||||||
|
|
||||||
## [34/48] 请对下面的程序文件做一个概述: request_llms\bridge_all.py
|
|
||||||
|
|
||||||
该文件包含两个函数：predict和predict_no_ui_long_connection，用于基于不同的LLM模型进行对话。该文件还包含一个lazyloadTiktoken类和一个LLM_CATCH_EXCEPTION修饰器函数。其中lazyloadTiktoken类用于懒加载模型的tokenizer，LLM_CATCH_EXCEPTION用于错误处理。整个文件还定义了一些全局变量和模型信息字典，用于引用和配置LLM模型。
|
|
||||||
|
|
||||||
## [35/48] 请对下面的程序文件做一个概述: request_llms\bridge_chatglm.py
|
|
||||||
|
|
||||||
这是一个Python程序文件，名为`bridge_chatglm.py`，其中定义了一个名为`GetGLMHandle`的类和三个方法：`predict_no_ui_long_connection`、 `predict`和 `stream_chat`。该文件依赖于多个Python库，如`transformers`和`sentencepiece`。该文件实现了一个聊天机器人，使用ChatGLM模型来生成回复，支持单线程和多线程方式。程序启动时需要加载ChatGLM的模型和tokenizer，需要一段时间。在配置文件`config.py`中设置参数会影响模型的内存和显存使用，因此程序可能会导致低配计算机卡死。
|
|
||||||
|
|
||||||
## [36/48] 请对下面的程序文件做一个概述: request_llms\bridge_chatgpt.py
|
|
||||||
|
|
||||||
该文件为 Python 代码文件，文件名为 request_llms\bridge_chatgpt.py。该代码文件主要提供三个函数：predict、predict_no_ui和 predict_no_ui_long_connection，用于发送至 chatGPT 并等待回复，获取输出。该代码文件还包含一些辅助函数，用于处理连接异常、生成 HTTP 请求等。该文件的代码架构清晰，使用了多个自定义函数和模块。
|
|
||||||
|
|
||||||
## [37/48] 请对下面的程序文件做一个概述: request_llms\bridge_jittorllms_llama.py
|
|
||||||
|
|
||||||
该代码文件实现了一个聊天机器人，其中使用了 JittorLLMs 模型。主要包括以下几个部分：
|
|
||||||
1. GetGLMHandle 类：一个进程类，用于加载 JittorLLMs 模型并接收并处理请求。
|
|
||||||
2. predict_no_ui_long_connection 函数：一个多线程方法，用于在后台运行聊天机器人。
|
|
||||||
3. predict 函数：一个单线程方法，用于在前端页面上交互式调用聊天机器人，以获取用户输入并返回相应的回复。
|
|
||||||
|
|
||||||
这个文件中还有一些辅助函数和全局变量，例如 importlib、time、threading 等。
|
|
||||||
|
|
||||||
## [38/48] 请对下面的程序文件做一个概述: request_llms\bridge_jittorllms_pangualpha.py
|
|
||||||
|
|
||||||
这个文件是为了实现使用jittorllms（一种机器学习模型）来进行聊天功能的代码。其中包括了模型加载、模型的参数加载、消息的收发等相关操作。其中使用了多进程和多线程来提高性能和效率。代码中还包括了处理依赖关系的函数和预处理函数等。
|
|
||||||
|
|
||||||
## [39/48] 请对下面的程序文件做一个概述: request_llms\bridge_jittorllms_rwkv.py
|
|
||||||
|
|
||||||
这个文件是一个Python程序，文件名为request_llm\bridge_jittorllms_rwkv.py。它依赖transformers、time、threading、importlib、multiprocessing等库。在文件中，通过定义GetGLMHandle类加载jittorllms模型参数和定义stream_chat方法来实现与jittorllms模型的交互。同时，该文件还定义了predict_no_ui_long_connection和predict方法来处理历史信息、调用jittorllms模型、接收回复信息并输出结果。
|
|
||||||
|
|
||||||
## [40/48] 请对下面的程序文件做一个概述: request_llms\bridge_moss.py
|
|
||||||
|
|
||||||
该文件为一个Python源代码文件，文件名为 request_llms\bridge_moss.py。代码定义了一个 GetGLMHandle 类和两个函数 predict_no_ui_long_connection 和 predict。
|
|
||||||
|
|
||||||
GetGLMHandle 类继承自Process类（多进程），主要功能是启动一个子进程并加载 MOSS 模型参数，通过 Pipe 进行主子进程的通信。该类还定义了 check_dependency、moss_init、run 和 stream_chat 等方法，其中 check_dependency 和 moss_init 是子进程的初始化方法，run 是子进程运行方法，stream_chat 实现了主进程和子进程的交互过程。
|
|
||||||
|
|
||||||
函数 predict_no_ui_long_connection 是多线程方法，调用 GetGLMHandle 类加载 MOSS 参数后使用 stream_chat 实现主进程和子进程的交互过程。
|
|
||||||
|
|
||||||
函数 predict 是单线程方法，通过调用 update_ui 将交互过程中 MOSS 的回复实时更新到UI（User Interface）中，并执行一个 named function（additional_fn）指定的函数对输入进行预处理。
|
|
||||||
|
|
||||||
## [41/48] 请对下面的程序文件做一个概述: request_llms\bridge_newbing.py
|
|
||||||
|
|
||||||
这是一个名为`bridge_newbing.py`的程序文件，包含三个部分：
|
|
||||||
|
|
||||||
第一部分使用from语句导入了`edge_gpt`模块的`NewbingChatbot`类。
|
|
||||||
|
|
||||||
第二部分定义了一个名为`NewBingHandle`的继承自进程类的子类，该类会检查依赖性并启动进程。同时，该部分还定义了一个名为`predict_no_ui_long_connection`的多线程方法和一个名为`predict`的单线程方法，用于与NewBing进行通信。
|
|
||||||
|
|
||||||
第三部分定义了一个名为`newbing_handle`的全局变量，并导出了`predict_no_ui_long_connection`和`predict`这两个方法，以供其他程序可以调用。
|
|
||||||
|
|
||||||
## [42/48] 请对下面的程序文件做一个概述: request_llms\bridge_newbingfree.py
|
|
||||||
|
|
||||||
这个Python文件包含了三部分内容。第一部分是来自edge_gpt_free.py文件的聊天机器人程序。第二部分是子进程Worker，用于调用主体。第三部分提供了两个函数：predict_no_ui_long_connection和predict用于调用NewBing聊天机器人和返回响应。其中predict函数还提供了一些参数用于控制聊天机器人的回复和更新UI界面。
|
|
||||||
|
|
||||||
## [43/48] 请对下面的程序文件做一个概述: request_llms\bridge_stackclaude.py
|
|
||||||
|
|
||||||
这是一个Python源代码文件，文件名为request_llm\bridge_stackclaude.py。代码分为三个主要部分：
|
|
||||||
|
|
||||||
第一部分定义了Slack API Client类，实现Slack消息的发送、接收、循环监听，用于与Slack API进行交互。
|
|
||||||
|
|
||||||
第二部分定义了ClaudeHandle类，继承Process类，用于创建子进程Worker，调用主体，实现Claude与用户交互的功能。
|
|
||||||
|
|
||||||
第三部分定义了predict_no_ui_long_connection和predict两个函数，主要用于通过调用ClaudeHandle对象的stream_chat方法来获取Claude的回复，并更新ui以显示相关信息。其中predict函数采用单线程方法，而predict_no_ui_long_connection函数使用多线程方法。
|
|
||||||
|
|
||||||
## [44/48] 请对下面的程序文件做一个概述: request_llms\bridge_tgui.py
|
|
||||||
|
|
||||||
该文件是一个Python代码文件，名为request_llm\bridge_tgui.py。它包含了一些函数用于与chatbot UI交互，并通过WebSocket协议与远程LLM模型通信完成文本生成任务，其中最重要的函数是predict()和predict_no_ui_long_connection()。这个程序还有其他的辅助函数，如random_hash()。整个代码文件在协作的基础上完成了一次修改。
|
|
||||||
|
|
||||||
## [45/48] 请对下面的程序文件做一个概述: request_llms\edge_gpt.py
|
|
||||||
|
|
||||||
该文件是一个用于调用Bing chatbot API的Python程序，它由多个类和辅助函数构成，可以根据给定的对话连接在对话中提出问题，使用websocket与远程服务通信。程序实现了一个聊天机器人，可以为用户提供人工智能聊天。
|
|
||||||
|
|
||||||
## [46/48] 请对下面的程序文件做一个概述: request_llms\edge_gpt_free.py
|
|
||||||
|
|
||||||
该代码文件为一个会话API，可通过Chathub发送消息以返回响应。其中使用了 aiohttp 和 httpx 库进行网络请求并发送。代码中包含了一些函数和常量，多数用于生成请求数据或是请求头信息等。同时该代码文件还包含了一个 Conversation 类，调用该类可实现对话交互。
|
|
||||||
|
|
||||||
## [47/48] 请对下面的程序文件做一个概述: request_llms\test_llms.py
|
|
||||||
|
|
||||||
这个文件是用于对llm模型进行单元测试的Python程序。程序导入一个名为"request_llms.bridge_newbingfree"的模块，然后三次使用该模块中的predict_no_ui_long_connection()函数进行预测，并输出结果。此外，还有一些注释掉的代码段，这些代码段也是关于模型预测的。
|
|
||||||
|
|
||||||
## 用一张Markdown表格简要描述以下文件的功能：
|
|
||||||
check_proxy.py, colorful.py, config.py, config_private.py, core_functional.py, crazy_functional.py, main.py, multi_language.py, theme.py, toolbox.py, crazy_functions\crazy_functions_test.py, crazy_functions\crazy_utils.py, crazy_functions\Latex全文润色.py, crazy_functions\Latex全文翻译.py, crazy_functions\__init__.py, crazy_functions\下载arxiv论文翻译摘要.py。根据以上分析，用一句话概括程序的整体功能。
|
|
||||||
|
|
||||||
| 文件名 | 功能描述 |
|
|
||||||
| ------ | ------ |
|
|
||||||
| check_proxy.py | 检查代理有效性及地理位置 |
|
|
||||||
| colorful.py | 控制台打印彩色文字 |
|
|
||||||
| config.py | 配置和参数设置 |
|
|
||||||
| config_private.py | 私人配置和参数设置 |
|
|
||||||
| core_functional.py | 核心函数和参数设置 |
|
|
||||||
| crazy_functional.py | 高级功能插件集合 |
|
|
||||||
| main.py | 一个 Chatbot 程序，提供各种学术翻译、文本处理和其他查询服务 |
|
|
||||||
| multi_language.py | 识别和翻译不同语言 |
|
|
||||||
| theme.py | 自定义 gradio 应用程序主题 |
|
|
||||||
| toolbox.py | 工具类库，用于协助实现各种功能 |
|
|
||||||
| crazy_functions\crazy_functions_test.py | 测试 crazy_functions 中的各种函数 |
|
|
||||||
| crazy_functions\crazy_utils.py | 工具函数，用于字符串处理、异常检测、Markdown 格式转换等 |
|
|
||||||
| crazy_functions\Latex全文润色.py | 对整个 Latex 项目进行润色和纠错 |
|
|
||||||
| crazy_functions\Latex全文翻译.py | 对整个 Latex 项目进行翻译 |
|
|
||||||
| crazy_functions\__init__.py | 模块初始化文件，标识 `crazy_functions` 是一个包 |
|
|
||||||
| crazy_functions\下载arxiv论文翻译摘要.py | 下载 `arxiv` 论文的 PDF 文件，并提取摘要和翻译 |
|
|
||||||
|
|
||||||
这些程序源文件提供了基础的文本和语言处理功能、工具函数和高级插件，使 Chatbot 能够处理各种复杂的学术文本问题，包括润色、翻译、搜索、下载、解析等。
|
|
||||||
|
|
||||||
## 用一张Markdown表格简要描述以下文件的功能：
|
|
||||||
crazy_functions\代码重写为全英文_多线程.py, crazy_functions\图片生成.py, crazy_functions\Conversation_To_File.py, crazy_functions\总结word文档.py, crazy_functions\总结音视频.py, crazy_functions\Markdown_Translate.py, crazy_functions\批量总结PDF文档.py, crazy_functions\批量总结PDF文档pdfminer.py, crazy_functions\PDF_Translate.py, crazy_functions\理解PDF文档内容.py, crazy_functions\生成函数注释.py, crazy_functions\联网的ChatGPT.py, crazy_functions\解析JupyterNotebook.py, crazy_functions\解析项目源代码.py, crazy_functions\询问多个大语言模型.py, crazy_functions\读文章写摘要.py。根据以上分析，用一句话概括程序的整体功能。
|
|
||||||
|
|
||||||
| 文件名 | 功能简述 |
|
|
||||||
| --- | --- |
|
|
||||||
| 代码重写为全英文_多线程.py | 将Python源代码文件中的中文内容转化为英文 |
|
|
||||||
| 图片生成.py | 根据激励文本使用GPT模型生成相应的图像 |
|
|
||||||
| Conversation_To_File.py | 将每次对话记录写入Markdown格式的文件中 |
|
|
||||||
| 总结word文档.py | 对输入的word文档进行摘要生成 |
|
|
||||||
| 总结音视频.py | 对输入的音视频文件进行摘要生成 |
|
|
||||||
| Markdown_Translate.py | 将指定目录下的Markdown文件进行中英文翻译 |
|
|
||||||
| 批量总结PDF文档.py | 对PDF文件进行切割和摘要生成 |
|
|
||||||
| 批量总结PDF文档pdfminer.py | 对PDF文件进行文本内容的提取和摘要生成 |
|
|
||||||
| PDF_Translate.py | 将指定目录下的PDF文件进行中英文翻译 |
|
|
||||||
| 理解PDF文档内容.py | 对PDF文件进行摘要生成和问题解答 |
|
|
||||||
| 生成函数注释.py | 自动生成Python函数的注释 |
|
|
||||||
| 联网的ChatGPT.py | 使用网络爬虫和ChatGPT模型进行聊天回答 |
|
|
||||||
| 解析JupyterNotebook.py | 对Jupyter Notebook进行代码解析 |
|
|
||||||
| 解析项目源代码.py | 对指定编程语言的源代码进行解析 |
|
|
||||||
| 询问多个大语言模型.py | 使用多个大语言模型对输入进行处理和回复 |
|
|
||||||
| 读文章写摘要.py | 对论文进行解析和全文摘要生成 |
|
|
||||||
|
|
||||||
概括程序的整体功能：提供了一系列处理文本、文件和代码的功能，使用了各类语言模型、多线程、网络请求和数据解析技术来提高效率和精度。
|
|
||||||
|
|
||||||
## 用一张Markdown表格简要描述以下文件的功能：
|
|
||||||
crazy_functions\谷歌检索小助手.py, crazy_functions\高级功能函数模板.py, request_llms\bridge_all.py, request_llms\bridge_chatglm.py, request_llms\bridge_chatgpt.py, request_llms\bridge_jittorllms_llama.py, request_llms\bridge_jittorllms_pangualpha.py, request_llms\bridge_jittorllms_rwkv.py, request_llms\bridge_moss.py, request_llms\bridge_newbing.py, request_llms\bridge_newbingfree.py, request_llms\bridge_stackclaude.py, request_llms\bridge_tgui.py, request_llms\edge_gpt.py, request_llms\edge_gpt_free.py, request_llms\test_llms.py。根据以上分析，用一句话概括程序的整体功能。
|
|
||||||
|
|
||||||
| 文件名 | 功能描述 |
|
|
||||||
| --- | --- |
|
|
||||||
| crazy_functions\谷歌检索小助手.py | 提供谷歌学术搜索页面中相关文章的元数据信息。 |
|
|
||||||
| crazy_functions\高级功能函数模板.py | 使用Unsplash API发送相关图片以回复用户的输入。 |
|
|
||||||
| request_llms\bridge_all.py | 基于不同LLM模型进行对话。 |
|
|
||||||
| request_llms\bridge_chatglm.py | 使用ChatGLM模型生成回复，支持单线程和多线程方式。 |
|
|
||||||
| request_llms\bridge_chatgpt.py | 基于GPT模型完成对话。 |
|
|
||||||
| request_llms\bridge_jittorllms_llama.py | 使用JittorLLMs模型完成对话，支持单线程和多线程方式。 |
|
|
||||||
| request_llms\bridge_jittorllms_pangualpha.py | 使用JittorLLMs模型完成对话，基于多进程和多线程方式。 |
|
|
||||||
| request_llms\bridge_jittorllms_rwkv.py | 使用JittorLLMs模型完成聊天功能，提供包括历史信息、参数调节等在内的多个功能选项。 |
|
|
||||||
| request_llms\bridge_moss.py | 加载Moss模型完成对话功能。 |
|
|
||||||
| request_llms\bridge_newbing.py | 使用Newbing聊天机器人进行对话，支持单线程和多线程方式。 |
|
|
||||||
| request_llms\bridge_newbingfree.py | 基于Bing chatbot API实现聊天机器人的文本生成功能。 |
|
|
||||||
| request_llms\bridge_stackclaude.py | 基于Slack API实现Claude与用户的交互。 |
|
|
||||||
| request_llms\bridge_tgui.py | 通过websocket实现聊天机器人与UI界面交互。 |
|
|
||||||
| request_llms\edge_gpt.py | 调用Bing chatbot API提供聊天机器人服务。 |
|
|
||||||
| request_llms\edge_gpt_free.py | 实现聊天机器人API，采用aiohttp和httpx工具库。 |
|
|
||||||
| request_llms\test_llms.py | 对llm模型进行单元测试。 |
|
|
||||||
| 程序整体功能 | 实现不同种类的聊天机器人，可以根据输入进行文本生成。 |
|
|
||||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -1,129 +0,0 @@
|
|||||||
{
|
|
||||||
"解析JupyterNotebook": "ParsingJupyterNotebook",
|
|
||||||
"Latex翻译中文并重新编译PDF": "TranslateChineseToEnglishInLatexAndRecompilePDF",
|
|
||||||
"联网的ChatGPT_bing版": "OnlineChatGPT_BingEdition",
|
|
||||||
"理解PDF文档内容标准文件输入": "UnderstandPdfDocumentContentStandardFileInput",
|
|
||||||
"Latex英文纠错加PDF对比": "CorrectEnglishInLatexWithPDFComparison",
|
|
||||||
"下载arxiv论文并翻译摘要": "DownloadArxivPaperAndTranslateAbstract",
|
|
||||||
"Markdown翻译指定语言": "TranslateMarkdownToSpecifiedLanguage",
|
|
||||||
"下载arxiv论文翻译摘要": "DownloadArxivPaperTranslateAbstract",
|
|
||||||
"解析一个Python项目": "ParsePythonProject",
|
|
||||||
"解析一个Golang项目": "ParseGolangProject",
|
|
||||||
"代码重写为全英文_多线程": "RewriteCodeToEnglish_MultiThreaded",
|
|
||||||
"解析一个CSharp项目": "ParsingCSharpProject",
|
|
||||||
"删除所有本地对话历史记录": "DeleteAllLocalConversationHistoryRecords",
|
|
||||||
"连接bing搜索回答问题": "ConnectBingSearchAnswerQuestion",
|
|
||||||
"Langchain知识库": "LangchainKnowledgeBase",
|
|
||||||
"把字符太少的块清除为回车": "ClearBlocksWithTooFewCharactersToNewline",
|
|
||||||
"Latex精细分解与转化": "DecomposeAndConvertLatex",
|
|
||||||
"解析一个C项目的头文件": "ParseCProjectHeaderFiles",
|
|
||||||
"Markdown英译中": "TranslateMarkdownFromEnglishToChinese",
|
|
||||||
"Markdown中译英": "MarkdownChineseToEnglish",
|
|
||||||
"数学动画生成manim": "MathematicalAnimationGenerationManim",
|
|
||||||
"chatglm微调工具": "ChatGLMFineTuningTool",
|
|
||||||
"解析一个Rust项目": "ParseRustProject",
|
|
||||||
"解析一个Java项目": "ParseJavaProject",
|
|
||||||
"联网的ChatGPT": "ChatGPTConnectedToNetwork",
|
|
||||||
"解析任意code项目": "ParseAnyCodeProject",
|
|
||||||
"合并小写开头的段落块": "MergeLowercaseStartingParagraphBlocks",
|
|
||||||
"Latex英文润色": "EnglishProofreadingForLatex",
|
|
||||||
"Latex全文润色": "FullTextProofreadingForLatex",
|
|
||||||
"询问多个大语言模型": "InquiryMultipleLargeLanguageModels",
|
|
||||||
"解析一个Lua项目": "ParsingLuaProject",
|
|
||||||
"解析ipynb文件": "ParsingIpynbFiles",
|
|
||||||
"批量总结PDF文档": "BatchSummarizePDFDocuments",
|
|
||||||
"批量翻译PDF文档": "BatchTranslatePDFDocuments",
|
|
||||||
"理解PDF文档内容": "UnderstandPdfDocumentContent",
|
|
||||||
"Latex中文润色": "LatexChineseProofreading",
|
|
||||||
"Latex英文纠错": "LatexEnglishCorrection",
|
|
||||||
"Latex全文翻译": "LatexFullTextTranslation",
|
|
||||||
"同时问询_指定模型": "InquireSimultaneously_SpecifiedModel",
|
|
||||||
"批量生成函数注释": "BatchGenerateFunctionComments",
|
|
||||||
"解析一个前端项目": "ParseFrontendProject",
|
|
||||||
"高阶功能模板函数": "HighOrderFunctionTemplateFunctions",
|
|
||||||
"高级功能函数模板": "AdvancedFunctionTemplate",
|
|
||||||
"总结word文档": "SummarizingWordDocuments",
|
|
||||||
"载入Conversation_To_File": "LoadConversationHistoryArchive",
|
|
||||||
"Latex中译英": "LatexChineseToEnglish",
|
|
||||||
"Latex英译中": "LatexEnglishToChinese",
|
|
||||||
"连接网络回答问题": "ConnectToNetworkToAnswerQuestions",
|
|
||||||
"交互功能模板函数": "InteractiveFunctionTemplateFunction",
|
|
||||||
"交互功能函数模板": "InteractiveFunctionFunctionTemplate",
|
|
||||||
"sprint亮靛": "SprintIndigo",
|
|
||||||
"print亮黄": "PrintBrightYellow",
|
|
||||||
"print亮绿": "PrintBrightGreen",
|
|
||||||
"print亮红": "PrintBrightRed",
|
|
||||||
"解析项目源代码": "ParseProjectSourceCode",
|
|
||||||
"解析一个C项目": "ParseCProject",
|
|
||||||
"全项目切换英文": "SwitchToEnglishForTheWholeProject",
|
|
||||||
"谷歌检索小助手": "GoogleSearchAssistant",
|
|
||||||
"读取知识库作答": "ReadKnowledgeArchiveAnswerQuestions",
|
|
||||||
"print亮蓝": "PrintBrightBlue",
|
|
||||||
"微调数据集生成": "FineTuneDatasetGeneration",
|
|
||||||
"清理多余的空行": "CleanUpExcessBlankLines",
|
|
||||||
"编译Latex": "CompileLatex",
|
|
||||||
"解析Paper": "ParsePaper",
|
|
||||||
"ipynb解释": "IpynbExplanation",
|
|
||||||
"读文章写摘要": "ReadArticleWriteSummary",
|
|
||||||
"生成函数注释": "GenerateFunctionComments",
|
|
||||||
"解析项目本身": "ParseProjectItself",
|
|
||||||
"专业词汇声明": "ProfessionalTerminologyDeclaration",
|
|
||||||
"解析docx": "ParseDocx",
|
|
||||||
"解析源代码新": "ParsingSourceCodeNew",
|
|
||||||
"总结音视频": "SummaryAudioVideo",
|
|
||||||
"知识库问答": "UpdateKnowledgeArchive",
|
|
||||||
"多文件润色": "ProofreadMultipleFiles",
|
|
||||||
"多文件翻译": "TranslateMultipleFiles",
|
|
||||||
"解析PDF": "ParsePDF",
|
|
||||||
"同时问询": "SimultaneousInquiry",
|
|
||||||
"图片生成": "ImageGeneration",
|
|
||||||
"动画生成": "AnimationGeneration",
|
|
||||||
"语音助手": "VoiceAssistant",
|
|
||||||
"启动微调": "StartFineTuning",
|
|
||||||
"清除缓存": "ClearCache",
|
|
||||||
"辅助功能": "Accessibility",
|
|
||||||
"虚空终端": "VoidTerminal",
|
|
||||||
"解析PDF_基于GROBID": "ParsePDF_BasedOnGROBID",
|
|
||||||
"虚空终端主路由": "VoidTerminalMainRoute",
|
|
||||||
"批量翻译PDF文档_NOUGAT": "BatchTranslatePDFDocuments_NOUGAT",
|
|
||||||
"解析PDF_基于NOUGAT": "ParsePDF_NOUGAT",
|
|
||||||
"解析一个Matlab项目": "AnalyzeAMatlabProject",
|
|
||||||
"函数动态生成": "DynamicFunctionGeneration",
|
|
||||||
"多智能体终端": "MultiAgentTerminal",
|
|
||||||
"多智能体": "MultiAgent",
|
|
||||||
"图片生成_DALLE2": "ImageGeneration_DALLE2",
|
|
||||||
"图片生成_DALLE3": "ImageGeneration_DALLE3",
|
|
||||||
"图片修改_DALLE2": "ImageModification_DALLE2",
|
|
||||||
"生成多种Mermaid图表": "GenerateMultipleMermaidCharts",
|
|
||||||
"知识库文件注入": "InjectKnowledgeBaseFiles",
|
|
||||||
"PDF翻译中文并重新编译PDF": "TranslatePDFToChineseAndRecompilePDF",
|
|
||||||
"随机小游戏": "RandomMiniGame",
|
|
||||||
"互动小游戏": "InteractiveMiniGame",
|
|
||||||
"解析历史输入": "ParseHistoricalInput",
|
|
||||||
"高阶功能模板函数示意图": "HighOrderFunctionTemplateDiagram",
|
|
||||||
"载入对话历史存档": "LoadChatHistoryArchive",
|
|
||||||
"对话历史存档": "ChatHistoryArchive",
|
|
||||||
"解析PDF_DOC2X_转Latex": "ParsePDF_DOC2X_toLatex",
|
|
||||||
"解析PDF_基于DOC2X": "ParsePDF_basedDOC2X",
|
|
||||||
"解析PDF_简单拆解": "ParsePDF_simpleDecomposition",
|
|
||||||
"解析PDF_DOC2X_单文件": "ParsePDF_DOC2X_singleFile",
|
|
||||||
"注释Python项目": "CommentPythonProject",
|
|
||||||
"注释源代码": "CommentSourceCode",
|
|
||||||
"log亮黄": "log_yellow",
|
|
||||||
"log亮绿": "log_green",
|
|
||||||
"log亮红": "log_red",
|
|
||||||
"log亮紫": "log_purple",
|
|
||||||
"log亮蓝": "log_blue",
|
|
||||||
"Rag问答": "RagQA",
|
|
||||||
"sprint红": "sprint_red",
|
|
||||||
"sprint绿": "sprint_green",
|
|
||||||
"sprint黄": "sprint_yellow",
|
|
||||||
"sprint蓝": "sprint_blue",
|
|
||||||
"sprint紫": "sprint_purple",
|
|
||||||
"sprint靛": "sprint_indigo",
|
|
||||||
"sprint亮红": "sprint_bright_red",
|
|
||||||
"sprint亮绿": "sprint_bright_green",
|
|
||||||
"sprint亮黄": "sprint_bright_yellow",
|
|
||||||
"sprint亮蓝": "sprint_bright_blue",
|
|
||||||
"sprint亮紫": "sprint_bright_purple"
|
|
||||||
}
|
|
||||||
File diff suppressed because it is too large
Load Diff
@@ -1,63 +0,0 @@
|
|||||||
# 使用音频交互功能
|
|
||||||
|
|
||||||
|
|
||||||
## 1. 安装额外依赖
|
|
||||||
```
|
|
||||||
pip install --upgrade pyOpenSSL webrtcvad scipy git+https://github.com/aliyun/alibabacloud-nls-python-sdk.git
|
|
||||||
```
|
|
||||||
|
|
||||||
如果因为特色网络问题导致上述命令无法执行：
|
|
||||||
1. git clone alibabacloud-nls-python-sdk这个项目（或者直接前往Github对应网址下载压缩包）.
|
|
||||||
命令行输入： `git clone https://github.com/aliyun/alibabacloud-nls-python-sdk.git`
|
|
||||||
1. 进入alibabacloud-nls-python-sdk目录命令行输入：`python setup.py install`
|
|
||||||
|
|
||||||
|
|
||||||
## 2. 配置音频功能开关 和 阿里云APPKEY（config.py/config_private.py/环境变量）
|
|
||||||
|
|
||||||
- 注册阿里云账号
|
|
||||||
- 开通 智能语音交互 （有免费白嫖时长）
|
|
||||||
- 获取token和appkey
|
|
||||||
- 未来将逐步用其他更廉价的云服务取代阿里云
|
|
||||||
|
|
||||||
```
|
|
||||||
ENABLE_AUDIO = True
|
|
||||||
ALIYUN_TOKEN = "554a50fcd0bb476c8d07bb630e94d20c" # 此token已经失效
|
|
||||||
ALIYUN_APPKEY = "RoPlZrM88DnAFkZK" # 此appkey已经失效
|
|
||||||
```
|
|
||||||
|
|
||||||
参考 https://help.aliyun.com/document_detail/450255.html
|
|
||||||
先有阿里云开发者账号，登录之后，需要开通 智能语音交互 的功能，可以免费获得一个token，然后在 全部项目 中，创建一个项目，可以获得一个appkey.
|
|
||||||
|
|
||||||
- 进阶功能
|
|
||||||
进一步填写ALIYUN_ACCESSKEY和ALIYUN_SECRET实现自动获取ALIYUN_TOKEN
|
|
||||||
```
|
|
||||||
ALIYUN_APPKEY = "RoP1ZrM84DnAFkZK"
|
|
||||||
ALIYUN_TOKEN = ""
|
|
||||||
ALIYUN_ACCESSKEY = "LTAI5q6BrFUzoRXVGUWnekh1"
|
|
||||||
ALIYUN_SECRET = "eHmI20AVWIaQZ0CiTD2bGQVsaP9i68"
|
|
||||||
```
|
|
||||||
|
|
||||||
|
|
||||||
## 3.启动
|
|
||||||
|
|
||||||
启动gpt-academic `python main.py`
|
|
||||||
|
|
||||||
## 4.点击record from microphe，授权音频采集
|
|
||||||
|
|
||||||
I 如果需要监听自己说话（不监听电脑音频），直接在浏览器中选择对应的麦即可
|
|
||||||
|
|
||||||
II 如果需要监听电脑音频（不监听自己说话），需要安装`VB-Audio VoiceMeeter`，打开声音控制面板(sound control panel)
|
|
||||||
- 1 `[把电脑的所有外放声音用VoiceMeeter截留]` 在输出区（playback）选项卡，把VoiceMeeter Input虚拟设备set as default设为默认播放设备。
|
|
||||||
- 2 `[把截留的声音释放到gpt-academic]` 打开gpt-academic主界面，授权音频采集后，在浏览器地址栏或者类似的地方会出现一个麦克风图标，打开后，按照浏览器的提示，选择VoiceMeeter虚拟麦克风。然后刷新页面，重新授权音频采集。
|
|
||||||
- 3 `[把截留的声音同时释放到耳机或音响]` 完成第一步之后，您应处于听不到电脑声音的状态。为了在截获音频的同时，避免影响正常使用，请完成这最后一步配置。在声音控制面板(sound control panel)输入区（recording）选项卡，把VoiceMeeter Output虚拟设备set as default。双击进入VoiceMeeter Output虚拟设备的设置。
|
|
||||||
- 3-1 进入VoiceMeeter Output虚拟设备子菜单，打开listen选项卡。
|
|
||||||
- 3-2 勾选Listen to this device。
|
|
||||||
- 3-3 在playback through this device下拉菜单中选择你的正常耳机或音响。
|
|
||||||
|
|
||||||
III `[把特殊软件（如腾讯会议）的外放声音用VoiceMeeter截留]` 在完成步骤II的基础上，在特殊软件（如腾讯会议）中，打开声音菜单，选择扬声器VoiceMeeter Input，选择麦克风为正常耳机麦。
|
|
||||||
|
|
||||||
VI 两种音频监听模式切换时，需要刷新页面才有效。
|
|
||||||
|
|
||||||
VII 非localhost运行+非https情况下无法打开录音功能的坑：https://blog.csdn.net/weixin_39461487/article/details/109594434
|
|
||||||
|
|
||||||
## 5.点击函数插件区“实时音频采集” 或者其他音频交互功能
|
|
||||||
@@ -1,164 +0,0 @@
|
|||||||
# 微软Azure云接入指南
|
|
||||||
|
|
||||||
## 方法一（旧方法，只能接入一个Azure模型）
|
|
||||||
|
|
||||||
- 通过以下教程，获取AZURE_ENDPOINT，AZURE_API_KEY，AZURE_ENGINE，直接修改 config 配置即可。配置的修改方法见本项目wiki。
|
|
||||||
|
|
||||||
## 方法二（新方法，接入多个Azure模型，并支持动态切换）
|
|
||||||
|
|
||||||
- 在方法一的基础上，注册并获取多组 AZURE_ENDPOINT，AZURE_API_KEY，AZURE_ENGINE
|
|
||||||
- 修改config中的AZURE_CFG_ARRAY和AVAIL_LLM_MODELS配置项，按照格式填入多个Azure模型的配置，如下所示：
|
|
||||||
|
|
||||||
```
|
|
||||||
AZURE_CFG_ARRAY = {
|
|
||||||
"azure-gpt-3.5": # 第一个模型，azure模型必须以"azure-"开头，注意您还需要将"azure-gpt-3.5"加入AVAIL_LLM_MODELS（模型下拉菜单）
|
|
||||||
{
|
|
||||||
"AZURE_ENDPOINT": "https://你亲手写的api名称.openai.azure.com/",
|
|
||||||
"AZURE_API_KEY": "cccccccccccccccccccccccccccccccc",
|
|
||||||
"AZURE_ENGINE": "填入你亲手写的部署名1",
|
|
||||||
"AZURE_MODEL_MAX_TOKEN": 4096,
|
|
||||||
},
|
|
||||||
"azure-gpt-4": # 第二个模型，azure模型必须以"azure-"开头，注意您还需要将"azure-gpt-4"加入AVAIL_LLM_MODELS（模型下拉菜单）
|
|
||||||
{
|
|
||||||
"AZURE_ENDPOINT": "https://你亲手写的api名称.openai.azure.com/",
|
|
||||||
"AZURE_API_KEY": "dddddddddddddddddddddddddddddddd",
|
|
||||||
"AZURE_ENGINE": "填入你亲手写的部署名2",
|
|
||||||
"AZURE_MODEL_MAX_TOKEN": 8192,
|
|
||||||
},
|
|
||||||
"azure-gpt-3.5-16k": # 第三个模型，azure模型必须以"azure-"开头，注意您还需要将"azure-gpt-3.5-16k"加入AVAIL_LLM_MODELS（模型下拉菜单）
|
|
||||||
{
|
|
||||||
"AZURE_ENDPOINT": "https://你亲手写的api名称.openai.azure.com/",
|
|
||||||
"AZURE_API_KEY": "eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee",
|
|
||||||
"AZURE_ENGINE": "填入你亲手写的部署名3",
|
|
||||||
"AZURE_MODEL_MAX_TOKEN": 16384,
|
|
||||||
},
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
# 通过微软Azure云服务申请 Openai API
|
|
||||||
|
|
||||||
由于Openai和微软的关系，现在是可以通过微软的Azure云计算服务直接访问openai的api，免去了注册和网络的问题。
|
|
||||||
|
|
||||||
快速入门的官方文档的链接是：[快速入门 - 开始通过 Azure OpenAI 服务使用 ChatGPT 和 GPT-4 - Azure OpenAI Service | Microsoft Learn](https://learn.microsoft.com/zh-cn/azure/cognitive-services/openai/chatgpt-quickstart?pivots=programming-language-python)
|
|
||||||
|
|
||||||
# 申请API
|
|
||||||
|
|
||||||
按文档中的“先决条件”的介绍，出了编程的环境以外，还需要以下三个条件：
|
|
||||||
|
|
||||||
1. Azure账号并创建订阅
|
|
||||||
|
|
||||||
2. 为订阅添加Azure OpenAI 服务
|
|
||||||
|
|
||||||
3. 部署模型
|
|
||||||
|
|
||||||
## Azure账号并创建订阅
|
|
||||||
|
|
||||||
### Azure账号
|
|
||||||
|
|
||||||
创建Azure的账号时最好是有微软的账号，这样似乎更容易获得免费额度（第一个月的200美元，实测了一下，如果用一个刚注册的微软账号登录Azure的话，并没有这一个月的免费额度）。
|
|
||||||
|
|
||||||
创建Azure账号的网址是：[立即创建 Azure 免费帐户 | Microsoft Azure](https://azure.microsoft.com/zh-cn/free/)
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
打开网页后，点击 “免费开始使用” 会跳转到登录或注册页面，如果有微软的账户，直接登录即可，如果没有微软账户，那就需要到微软的网页再另行注册一个。
|
|
||||||
|
|
||||||
注意，Azure的页面和政策时不时会变化，已实际最新显示的为准就好。
|
|
||||||
|
|
||||||
### 创建订阅
|
|
||||||
|
|
||||||
注册好Azure后便可进入主页：
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
首先需要在订阅里进行添加操作，点开后即可进入订阅的页面：
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
第一次进来应该是空的，点添加即可创建新的订阅（可以是“免费”或者“即付即用”的订阅），其中订阅ID是后面申请Azure OpenAI需要使用的。
|
|
||||||
|
|
||||||
## 为订阅添加Azure OpenAI服务
|
|
||||||
|
|
||||||
之后回到首页，点Azure OpenAI即可进入OpenAI服务的页面（如果不显示的话，则在首页上方的搜索栏里搜索“openai”即可）。
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
不过现在这个服务还不能用。在使用前，还需要在这个网址申请一下：
|
|
||||||
|
|
||||||
[Request Access to Azure OpenAI Service (microsoft.com)](https://customervoice.microsoft.com/Pages/ResponsePage.aspx?id=v4j5cvGGr0GRqy180BHbR7en2Ais5pxKtso_Pz4b1_xUOFA5Qk1UWDRBMjg0WFhPMkIzTzhKQ1dWNyQlQCN0PWcu)
|
|
||||||
|
|
||||||
这里有二十来个问题，按照要求和自己的实际情况填写即可。
|
|
||||||
|
|
||||||
其中需要注意的是
|
|
||||||
|
|
||||||
1. 千万记得填对"订阅ID"
|
|
||||||
|
|
||||||
2. 需要填一个公司邮箱（可以不是注册用的邮箱）和公司网址
|
|
||||||
|
|
||||||
之后，在回到上面那个页面，点创建，就会进入创建页面了：
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
需要填入“资源组”和“名称”，按照自己的需要填入即可。
|
|
||||||
|
|
||||||
完成后，在主页的“资源”里就可以看到刚才创建的“资源”了，点击进入后，就可以进行最后的部署了。
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
## 部署模型
|
|
||||||
|
|
||||||
进入资源页面后，在部署模型前，可以先点击“开发”，把密钥和终结点记下来。
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
之后，就可以去部署模型了，点击“部署”即可，会跳转到 Azure OpenAI Stuido 进行下面的操作：
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
进入 Azure OpenAi Studio 后，点击新建部署，会弹出如下对话框：
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
在这里选 gpt-35-turbo 或需要的模型并按需要填入“部署名”即可完成模型的部署。
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
这个部署名需要记下来。
|
|
||||||
|
|
||||||
到现在为止，申请操作就完成了，需要记下来的有下面几个东西：
|
|
||||||
|
|
||||||
● 密钥（对应AZURE_API_KEY，1或2都可以）
|
|
||||||
|
|
||||||
● 终结点 （对应AZURE_ENDPOINT）
|
|
||||||
|
|
||||||
● 部署名（对应AZURE_ENGINE，不是模型名）
|
|
||||||
|
|
||||||
|
|
||||||
# 修改 config.py
|
|
||||||
|
|
||||||
```
|
|
||||||
LLM_MODEL = "azure-gpt-3.5" # 指定启动时的默认模型，当然事后从下拉菜单选也ok
|
|
||||||
|
|
||||||
AZURE_ENDPOINT = "填入终结点" # 见上述图片
|
|
||||||
AZURE_API_KEY = "填入azure openai api的密钥"
|
|
||||||
AZURE_API_VERSION = "2023-05-15" # 默认使用 2023-05-15 版本，无需修改
|
|
||||||
AZURE_ENGINE = "填入部署名" # 见上述图片
|
|
||||||
|
|
||||||
|
|
||||||
# 例如
|
|
||||||
API_KEY = '6424e9d19e674092815cea1cb35e67a5'
|
|
||||||
AZURE_ENDPOINT = 'https://rhtjjjjjj.openai.azure.com/'
|
|
||||||
AZURE_ENGINE = 'qqwe'
|
|
||||||
LLM_MODEL = "azure-gpt-3.5" # 可选 ↓↓↓
|
|
||||||
```
|
|
||||||
|
|
||||||
|
|
||||||
# 关于费用
|
|
||||||
|
|
||||||
Azure OpenAI API 还是需要一些费用的（免费订阅只有1个月有效期）
|
|
||||||
|
|
||||||
具体可以可以看这个网址 ：[Azure OpenAI 服务 - 定价| Microsoft Azure](https://azure.microsoft.com/zh-cn/pricing/details/cognitive-services/openai-service/?cdn=disable)
|
|
||||||
|
|
||||||
并非网上说的什么“一年白嫖”，但注册方法以及网络问题都比直接使用openai的api要简单一些。
|
|
||||||
@@ -1,58 +0,0 @@
|
|||||||
# 使用TTS文字转语音
|
|
||||||
|
|
||||||
|
|
||||||
## 1. 使用EDGE-TTS（简单）
|
|
||||||
|
|
||||||
将本项目配置项修改如下即可
|
|
||||||
|
|
||||||
```
|
|
||||||
TTS_TYPE = "EDGE_TTS"
|
|
||||||
EDGE_TTS_VOICE = "zh-CN-XiaoxiaoNeural"
|
|
||||||
```
|
|
||||||
|
|
||||||
## 2. 使用SoVITS（需要有显卡）
|
|
||||||
|
|
||||||
使用以下docker-compose.yml文件，先启动SoVITS服务API
|
|
||||||
|
|
||||||
1. 创建以下文件夹结构
|
|
||||||
```shell
|
|
||||||
.
|
|
||||||
├── docker-compose.yml
|
|
||||||
└── reference
|
|
||||||
├── clone_target_txt.txt
|
|
||||||
└── clone_target_wave.mp3
|
|
||||||
```
|
|
||||||
2. 其中`docker-compose.yml`为
|
|
||||||
```yaml
|
|
||||||
version: '3.8'
|
|
||||||
services:
|
|
||||||
gpt-sovits:
|
|
||||||
image: fuqingxu/sovits_gptac_trim:latest
|
|
||||||
container_name: sovits_gptac_container
|
|
||||||
working_dir: /workspace/gpt_sovits_demo
|
|
||||||
environment:
|
|
||||||
- is_half=False
|
|
||||||
- is_share=False
|
|
||||||
volumes:
|
|
||||||
- ./reference:/reference
|
|
||||||
ports:
|
|
||||||
- "19880:9880" # 19880 为 sovits api 的暴露端口，记住它
|
|
||||||
shm_size: 16G
|
|
||||||
deploy:
|
|
||||||
resources:
|
|
||||||
reservations:
|
|
||||||
devices:
|
|
||||||
- driver: nvidia
|
|
||||||
count: "all"
|
|
||||||
capabilities: [gpu]
|
|
||||||
command: bash -c "python3 api.py"
|
|
||||||
```
|
|
||||||
3. 其中`clone_target_wave.mp3`为需要克隆的角色音频，`clone_target_txt.txt`为该音频对应的文字文本（ https://wiki.biligame.com/ys/%E8%A7%92%E8%89%B2%E8%AF%AD%E9%9F%B3 ）
|
|
||||||
4. 运行`docker-compose up`
|
|
||||||
5. 将本项目配置项修改如下即可
|
|
||||||
(19880 为 sovits api 的暴露端口，与docker-compose.yml中的端口对应)
|
|
||||||
```
|
|
||||||
TTS_TYPE = "LOCAL_SOVITS_API"
|
|
||||||
GPT_SOVITS_URL = "http://127.0.0.1:19880"
|
|
||||||
```
|
|
||||||
6. 启动本项目
|
|
||||||
@@ -1,46 +0,0 @@
|
|||||||
# 使用VLLM
|
|
||||||
|
|
||||||
|
|
||||||
## 1. 首先启动 VLLM，自行选择模型
|
|
||||||
|
|
||||||
```
|
|
||||||
python -m vllm.entrypoints.openai.api_server --model /home/hmp/llm/cache/Qwen1___5-32B-Chat --tensor-parallel-size 2 --dtype=half
|
|
||||||
```
|
|
||||||
|
|
||||||
这里使用了存储在 `/home/hmp/llm/cache/Qwen1___5-32B-Chat` 的本地模型，可以根据自己的需求更改。
|
|
||||||
|
|
||||||
## 2. 测试 VLLM
|
|
||||||
|
|
||||||
```
|
|
||||||
curl http://localhost:8000/v1/chat/completions \
|
|
||||||
-H "Content-Type: application/json" \
|
|
||||||
-d '{
|
|
||||||
"model": "/home/hmp/llm/cache/Qwen1___5-32B-Chat",
|
|
||||||
"messages": [
|
|
||||||
{"role": "system", "content": "You are a helpful assistant."},
|
|
||||||
{"role": "user", "content": "怎么实现一个去中心化的控制器?"}
|
|
||||||
]
|
|
||||||
}'
|
|
||||||
```
|
|
||||||
|
|
||||||
## 3. 配置本项目
|
|
||||||
|
|
||||||
```
|
|
||||||
API_KEY = "sk-123456789xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx123456789"
|
|
||||||
LLM_MODEL = "vllm-/home/hmp/llm/cache/Qwen1___5-32B-Chat(max_token=4096)"
|
|
||||||
API_URL_REDIRECT = {"https://api.openai.com/v1/chat/completions": "http://localhost:8000/v1/chat/completions"}
|
|
||||||
```
|
|
||||||
|
|
||||||
```
|
|
||||||
"vllm-/home/hmp/llm/cache/Qwen1___5-32B-Chat(max_token=4096)"
|
|
||||||
其中
|
|
||||||
"vllm-" 是前缀（必要）
|
|
||||||
"/home/hmp/llm/cache/Qwen1___5-32B-Chat" 是模型名（必要）
|
|
||||||
"(max_token=6666)" 是配置（非必要）
|
|
||||||
```
|
|
||||||
|
|
||||||
## 4. 启动！
|
|
||||||
|
|
||||||
```
|
|
||||||
python main.py
|
|
||||||
```
|
|
||||||
34
main.py
34
main.py
@@ -34,7 +34,7 @@ def encode_plugin_info(k, plugin)->str:
|
|||||||
|
|
||||||
def main():
|
def main():
|
||||||
import gradio as gr
|
import gradio as gr
|
||||||
if gr.__version__ not in ['3.32.12']:
|
if gr.__version__ not in ['3.32.9', '3.32.10', '3.32.11']:
|
||||||
raise ModuleNotFoundError("使用项目内置Gradio获取最优体验! 请运行 `pip install -r requirements.txt` 指令安装内置Gradio及其他依赖, 详情信息见requirements.txt.")
|
raise ModuleNotFoundError("使用项目内置Gradio获取最优体验! 请运行 `pip install -r requirements.txt` 指令安装内置Gradio及其他依赖, 详情信息见requirements.txt.")
|
||||||
|
|
||||||
# 一些基础工具
|
# 一些基础工具
|
||||||
@@ -57,10 +57,10 @@ def main():
|
|||||||
# 如果WEB_PORT是-1, 则随机选取WEB端口
|
# 如果WEB_PORT是-1, 则随机选取WEB端口
|
||||||
PORT = find_free_port() if WEB_PORT <= 0 else WEB_PORT
|
PORT = find_free_port() if WEB_PORT <= 0 else WEB_PORT
|
||||||
from check_proxy import get_current_version
|
from check_proxy import get_current_version
|
||||||
from themes.theme import adjust_theme, advanced_css, theme_declaration, js_code_clear, js_code_show_or_hide, js_code_show_or_hide_group2
|
from themes.theme import adjust_theme, advanced_css, theme_declaration, js_code_clear, js_code_reset, js_code_show_or_hide, js_code_show_or_hide_group2
|
||||||
from themes.theme import js_code_for_toggle_darkmode
|
from themes.theme import js_code_for_toggle_darkmode, js_code_for_persistent_cookie_init
|
||||||
from themes.theme import load_dynamic_theme, to_cookie_str, from_cookie_str, assign_user_uuid
|
from themes.theme import load_dynamic_theme, to_cookie_str, from_cookie_str, assign_user_uuid
|
||||||
title_html = f"<h1 align=\"center\">GPT 学术优化 {get_current_version()}</h1>{theme_declaration}"
|
title_html = f"<h1 align=\"center\">蚂小财MVP测试 {get_current_version()}</h1>{theme_declaration}"
|
||||||
|
|
||||||
|
|
||||||
# 一些普通功能模块
|
# 一些普通功能模块
|
||||||
@@ -96,7 +96,7 @@ def main():
|
|||||||
customize_btns = {}
|
customize_btns = {}
|
||||||
predefined_btns = {}
|
predefined_btns = {}
|
||||||
from shared_utils.cookie_manager import make_cookie_cache, make_history_cache
|
from shared_utils.cookie_manager import make_cookie_cache, make_history_cache
|
||||||
with gr.Blocks(title="GPT 学术优化", theme=set_theme, analytics_enabled=False, css=advanced_css) as app_block:
|
with gr.Blocks(title="蚂小财MVP测试", theme=set_theme, analytics_enabled=False, css=advanced_css) as app_block:
|
||||||
gr.HTML(title_html)
|
gr.HTML(title_html)
|
||||||
secret_css = gr.Textbox(visible=False, elem_id="secret_css")
|
secret_css = gr.Textbox(visible=False, elem_id="secret_css")
|
||||||
register_advanced_plugin_init_arr = ""
|
register_advanced_plugin_init_arr = ""
|
||||||
@@ -106,18 +106,18 @@ def main():
|
|||||||
with gr_L2(scale=2, elem_id="gpt-chat"):
|
with gr_L2(scale=2, elem_id="gpt-chat"):
|
||||||
chatbot = gr.Chatbot(label=f"当前模型：{LLM_MODEL}", elem_id="gpt-chatbot")
|
chatbot = gr.Chatbot(label=f"当前模型：{LLM_MODEL}", elem_id="gpt-chatbot")
|
||||||
if LAYOUT == "TOP-DOWN": chatbot.style(height=CHATBOT_HEIGHT)
|
if LAYOUT == "TOP-DOWN": chatbot.style(height=CHATBOT_HEIGHT)
|
||||||
history, _, _ = make_history_cache() # 定义 后端state（history）、前端（history_cache）、后端setter（history_cache_update）三兄弟
|
history, history_cache, history_cache_update = make_history_cache() # 定义 后端state（history）、前端（history_cache）、后端setter（history_cache_update）三兄弟
|
||||||
with gr_L2(scale=1, elem_id="gpt-panel"):
|
with gr_L2(scale=1, elem_id="gpt-panel"):
|
||||||
with gr.Accordion("输入区", open=True, elem_id="input-panel") as area_input_primary:
|
with gr.Accordion("输入区", open=True, elem_id="input-panel") as area_input_primary:
|
||||||
with gr.Row():
|
with gr.Row():
|
||||||
txt = gr.Textbox(show_label=False, placeholder="Input question here.", elem_id='user_input_main').style(container=False)
|
txt = gr.Textbox(show_label=False, placeholder="Input question here.", elem_id='user_input_main').style(container=False)
|
||||||
with gr.Row(elem_id="gpt-submit-row"):
|
with gr.Row(elem_id="gpt-submit-row"):
|
||||||
multiplex_submit_btn = gr.Button("提交", elem_id="elem_submit_visible", variant="primary")
|
multiplex_submit_btn = gr.Button("蚂小财测试", elem_id="elem_submit_visible", variant="primary")
|
||||||
multiplex_sel = gr.Dropdown(
|
multiplex_sel = gr.Dropdown(
|
||||||
choices=get_multiplex_button_functions().keys(), value="常规对话",
|
choices=get_multiplex_button_functions().keys(), value="蚂小财测试",
|
||||||
interactive=True, label='', show_label=False,
|
interactive=True, label='', show_label=False,
|
||||||
elem_classes='normal_mut_select', elem_id="gpt-submit-dropdown").style(container=False)
|
elem_classes='normal_mut_select', elem_id="gpt-submit-dropdown").style(container=False)
|
||||||
submit_btn = gr.Button("提交", elem_id="elem_submit", variant="primary", visible=False)
|
submit_btn = gr.Button("蚂小财测试", elem_id="elem_submit", variant="primary", visible=False)
|
||||||
with gr.Row():
|
with gr.Row():
|
||||||
resetBtn = gr.Button("重置", elem_id="elem_reset", variant="secondary"); resetBtn.style(size="sm")
|
resetBtn = gr.Button("重置", elem_id="elem_reset", variant="secondary"); resetBtn.style(size="sm")
|
||||||
stopBtn = gr.Button("停止", elem_id="elem_stop", variant="secondary"); stopBtn.style(size="sm")
|
stopBtn = gr.Button("停止", elem_id="elem_stop", variant="secondary"); stopBtn.style(size="sm")
|
||||||
@@ -174,7 +174,6 @@ def main():
|
|||||||
with gr.Accordion("点击展开“文件下载区”。", open=False) as area_file_up:
|
with gr.Accordion("点击展开“文件下载区”。", open=False) as area_file_up:
|
||||||
file_upload = gr.Files(label="任何文件, 推荐上传压缩文件(zip, tar)", file_count="multiple", elem_id="elem_upload")
|
file_upload = gr.Files(label="任何文件, 推荐上传压缩文件(zip, tar)", file_count="multiple", elem_id="elem_upload")
|
||||||
|
|
||||||
|
|
||||||
# 左上角工具栏定义
|
# 左上角工具栏定义
|
||||||
from themes.gui_toolbar import define_gui_toolbar
|
from themes.gui_toolbar import define_gui_toolbar
|
||||||
checkboxes, checkboxes_2, max_length_sl, theme_dropdown, system_prompt, file_upload_2, md_dropdown, top_p, temperature = \
|
checkboxes, checkboxes_2, max_length_sl, theme_dropdown, system_prompt, file_upload_2, md_dropdown, top_p, temperature = \
|
||||||
@@ -185,9 +184,6 @@ def main():
|
|||||||
area_input_secondary, txt2, area_customize, _, resetBtn2, clearBtn2, stopBtn2 = \
|
area_input_secondary, txt2, area_customize, _, resetBtn2, clearBtn2, stopBtn2 = \
|
||||||
define_gui_floating_menu(customize_btns, functional, predefined_btns, cookies, web_cookie_cache)
|
define_gui_floating_menu(customize_btns, functional, predefined_btns, cookies, web_cookie_cache)
|
||||||
|
|
||||||
# 浮动时间线定义
|
|
||||||
gr.Spark()
|
|
||||||
|
|
||||||
# 插件二级菜单的实现
|
# 插件二级菜单的实现
|
||||||
from themes.gui_advanced_plugin_class import define_gui_advanced_plugin_class
|
from themes.gui_advanced_plugin_class import define_gui_advanced_plugin_class
|
||||||
new_plugin_callback, route_switchy_bt_with_arg, usr_confirmed_arg = \
|
new_plugin_callback, route_switchy_bt_with_arg, usr_confirmed_arg = \
|
||||||
@@ -226,11 +222,11 @@ def main():
|
|||||||
multiplex_sel.select(
|
multiplex_sel.select(
|
||||||
None, [multiplex_sel], None, _js=f"""(multiplex_sel)=>run_multiplex_shift(multiplex_sel)""")
|
None, [multiplex_sel], None, _js=f"""(multiplex_sel)=>run_multiplex_shift(multiplex_sel)""")
|
||||||
cancel_handles.append(submit_btn.click(**predict_args))
|
cancel_handles.append(submit_btn.click(**predict_args))
|
||||||
resetBtn.click(None, None, [chatbot, history, status], _js="""(a,b,c)=>clear_conversation(a,b,c)""") # 先在前端快速清除chatbot&status
|
resetBtn.click(None, None, [chatbot, history, status], _js=js_code_reset) # 先在前端快速清除chatbot&status
|
||||||
resetBtn2.click(None, None, [chatbot, history, status], _js="""(a,b,c)=>clear_conversation(a,b,c)""") # 先在前端快速清除chatbot&status
|
resetBtn2.click(None, None, [chatbot, history, status], _js=js_code_reset) # 先在前端快速清除chatbot&status
|
||||||
# reset_server_side_args = (lambda history: ([], [], "已重置"), [history], [chatbot, history, status])
|
reset_server_side_args = (lambda history: ([], [], "已重置", json.dumps(history)), [history], [chatbot, history, status, history_cache])
|
||||||
# resetBtn.click(*reset_server_side_args) # 再在后端清除history
|
resetBtn.click(*reset_server_side_args) # 再在后端清除history，把history转存history_cache备用
|
||||||
# resetBtn2.click(*reset_server_side_args) # 再在后端清除history
|
resetBtn2.click(*reset_server_side_args) # 再在后端清除history，把history转存history_cache备用
|
||||||
clearBtn.click(None, None, [txt, txt2], _js=js_code_clear)
|
clearBtn.click(None, None, [txt, txt2], _js=js_code_clear)
|
||||||
clearBtn2.click(None, None, [txt, txt2], _js=js_code_clear)
|
clearBtn2.click(None, None, [txt, txt2], _js=js_code_clear)
|
||||||
if AUTO_CLEAR_TXT:
|
if AUTO_CLEAR_TXT:
|
||||||
@@ -330,7 +326,7 @@ def main():
|
|||||||
from shared_utils.cookie_manager import load_web_cookie_cache__fn_builder
|
from shared_utils.cookie_manager import load_web_cookie_cache__fn_builder
|
||||||
load_web_cookie_cache = load_web_cookie_cache__fn_builder(customize_btns, cookies, predefined_btns)
|
load_web_cookie_cache = load_web_cookie_cache__fn_builder(customize_btns, cookies, predefined_btns)
|
||||||
app_block.load(load_web_cookie_cache, inputs = [web_cookie_cache, cookies],
|
app_block.load(load_web_cookie_cache, inputs = [web_cookie_cache, cookies],
|
||||||
outputs = [web_cookie_cache, cookies, *customize_btns.values(), *predefined_btns.values()], _js="""persistent_cookie_init""")
|
outputs = [web_cookie_cache, cookies, *customize_btns.values(), *predefined_btns.values()], _js=js_code_for_persistent_cookie_init)
|
||||||
app_block.load(None, inputs=[], outputs=None, _js=f"""()=>GptAcademicJavaScriptInit("{DARK_MODE}","{INIT_SYS_PROMPT}","{ADD_WAIFU}","{LAYOUT}","{TTS_TYPE}")""") # 配置暗色主题或亮色主题
|
app_block.load(None, inputs=[], outputs=None, _js=f"""()=>GptAcademicJavaScriptInit("{DARK_MODE}","{INIT_SYS_PROMPT}","{ADD_WAIFU}","{LAYOUT}","{TTS_TYPE}")""") # 配置暗色主题或亮色主题
|
||||||
app_block.load(None, inputs=[], outputs=None, _js="""()=>{REP}""".replace("REP", register_advanced_plugin_init_arr))
|
app_block.load(None, inputs=[], outputs=None, _js="""()=>{REP}""".replace("REP", register_advanced_plugin_init_arr))
|
||||||
|
|
||||||
|
|||||||
@@ -26,9 +26,6 @@ from .bridge_chatglm import predict as chatglm_ui
|
|||||||
from .bridge_chatglm3 import predict_no_ui_long_connection as chatglm3_noui
|
from .bridge_chatglm3 import predict_no_ui_long_connection as chatglm3_noui
|
||||||
from .bridge_chatglm3 import predict as chatglm3_ui
|
from .bridge_chatglm3 import predict as chatglm3_ui
|
||||||
|
|
||||||
from .bridge_chatglm4 import predict_no_ui_long_connection as chatglm4_noui
|
|
||||||
from .bridge_chatglm4 import predict as chatglm4_ui
|
|
||||||
|
|
||||||
from .bridge_qianfan import predict_no_ui_long_connection as qianfan_noui
|
from .bridge_qianfan import predict_no_ui_long_connection as qianfan_noui
|
||||||
from .bridge_qianfan import predict as qianfan_ui
|
from .bridge_qianfan import predict as qianfan_ui
|
||||||
|
|
||||||
@@ -79,7 +76,6 @@ cohere_endpoint = "https://api.cohere.ai/v1/chat"
|
|||||||
ollama_endpoint = "http://localhost:11434/api/chat"
|
ollama_endpoint = "http://localhost:11434/api/chat"
|
||||||
yimodel_endpoint = "https://api.lingyiwanwu.com/v1/chat/completions"
|
yimodel_endpoint = "https://api.lingyiwanwu.com/v1/chat/completions"
|
||||||
deepseekapi_endpoint = "https://api.deepseek.com/v1/chat/completions"
|
deepseekapi_endpoint = "https://api.deepseek.com/v1/chat/completions"
|
||||||
grok_model_endpoint = "https://api.x.ai/v1/chat/completions"
|
|
||||||
|
|
||||||
if not AZURE_ENDPOINT.endswith('/'): AZURE_ENDPOINT += '/'
|
if not AZURE_ENDPOINT.endswith('/'): AZURE_ENDPOINT += '/'
|
||||||
azure_endpoint = AZURE_ENDPOINT + f'openai/deployments/{AZURE_ENGINE}/chat/completions?api-version=2023-05-15'
|
azure_endpoint = AZURE_ENDPOINT + f'openai/deployments/{AZURE_ENGINE}/chat/completions?api-version=2023-05-15'
|
||||||
@@ -101,7 +97,6 @@ if cohere_endpoint in API_URL_REDIRECT: cohere_endpoint = API_URL_REDIRECT[coher
|
|||||||
if ollama_endpoint in API_URL_REDIRECT: ollama_endpoint = API_URL_REDIRECT[ollama_endpoint]
|
if ollama_endpoint in API_URL_REDIRECT: ollama_endpoint = API_URL_REDIRECT[ollama_endpoint]
|
||||||
if yimodel_endpoint in API_URL_REDIRECT: yimodel_endpoint = API_URL_REDIRECT[yimodel_endpoint]
|
if yimodel_endpoint in API_URL_REDIRECT: yimodel_endpoint = API_URL_REDIRECT[yimodel_endpoint]
|
||||||
if deepseekapi_endpoint in API_URL_REDIRECT: deepseekapi_endpoint = API_URL_REDIRECT[deepseekapi_endpoint]
|
if deepseekapi_endpoint in API_URL_REDIRECT: deepseekapi_endpoint = API_URL_REDIRECT[deepseekapi_endpoint]
|
||||||
if grok_model_endpoint in API_URL_REDIRECT: grok_model_endpoint = API_URL_REDIRECT[grok_model_endpoint]
|
|
||||||
|
|
||||||
# 获取tokenizer
|
# 获取tokenizer
|
||||||
tokenizer_gpt35 = LazyloadTiktoken("gpt-3.5-turbo")
|
tokenizer_gpt35 = LazyloadTiktoken("gpt-3.5-turbo")
|
||||||
@@ -419,7 +414,6 @@ model_info = {
|
|||||||
"token_cnt": get_token_num_gpt4,
|
"token_cnt": get_token_num_gpt4,
|
||||||
},
|
},
|
||||||
|
|
||||||
# ChatGLM本地模型
|
|
||||||
# 将 chatglm 直接对齐到 chatglm2
|
# 将 chatglm 直接对齐到 chatglm2
|
||||||
"chatglm": {
|
"chatglm": {
|
||||||
"fn_with_ui": chatglm_ui,
|
"fn_with_ui": chatglm_ui,
|
||||||
@@ -445,14 +439,6 @@ model_info = {
|
|||||||
"tokenizer": tokenizer_gpt35,
|
"tokenizer": tokenizer_gpt35,
|
||||||
"token_cnt": get_token_num_gpt35,
|
"token_cnt": get_token_num_gpt35,
|
||||||
},
|
},
|
||||||
"chatglm4": {
|
|
||||||
"fn_with_ui": chatglm4_ui,
|
|
||||||
"fn_without_ui": chatglm4_noui,
|
|
||||||
"endpoint": None,
|
|
||||||
"max_token": 8192,
|
|
||||||
"tokenizer": tokenizer_gpt35,
|
|
||||||
"token_cnt": get_token_num_gpt35,
|
|
||||||
},
|
|
||||||
"qianfan": {
|
"qianfan": {
|
||||||
"fn_with_ui": qianfan_ui,
|
"fn_with_ui": qianfan_ui,
|
||||||
"fn_without_ui": qianfan_noui,
|
"fn_without_ui": qianfan_noui,
|
||||||
@@ -900,31 +886,6 @@ if any(item in yi_models for item in AVAIL_LLM_MODELS):
|
|||||||
})
|
})
|
||||||
except:
|
except:
|
||||||
logger.error(trimmed_format_exc())
|
logger.error(trimmed_format_exc())
|
||||||
|
|
||||||
|
|
||||||
# -=-=-=-=-=-=- Grok model from x.ai -=-=-=-=-=-=-
|
|
||||||
grok_models = ["grok-beta"]
|
|
||||||
if any(item in grok_models for item in AVAIL_LLM_MODELS):
|
|
||||||
try:
|
|
||||||
grok_beta_128k_noui, grok_beta_128k_ui = get_predict_function(
|
|
||||||
api_key_conf_name="GROK_API_KEY", max_output_token=8192, disable_proxy=False
|
|
||||||
)
|
|
||||||
|
|
||||||
model_info.update({
|
|
||||||
"grok-beta": {
|
|
||||||
"fn_with_ui": grok_beta_128k_ui,
|
|
||||||
"fn_without_ui": grok_beta_128k_noui,
|
|
||||||
"can_multi_thread": True,
|
|
||||||
"endpoint": grok_model_endpoint,
|
|
||||||
"max_token": 128000,
|
|
||||||
"tokenizer": tokenizer_gpt35,
|
|
||||||
"token_cnt": get_token_num_gpt35,
|
|
||||||
},
|
|
||||||
|
|
||||||
})
|
|
||||||
except:
|
|
||||||
logger.error(trimmed_format_exc())
|
|
||||||
|
|
||||||
# -=-=-=-=-=-=- 讯飞星火认知大模型 -=-=-=-=-=-=-
|
# -=-=-=-=-=-=- 讯飞星火认知大模型 -=-=-=-=-=-=-
|
||||||
if "spark" in AVAIL_LLM_MODELS:
|
if "spark" in AVAIL_LLM_MODELS:
|
||||||
try:
|
try:
|
||||||
|
|||||||
@@ -23,33 +23,39 @@ class GetGLM3Handle(LocalLLMHandle):
|
|||||||
import os
|
import os
|
||||||
import platform
|
import platform
|
||||||
|
|
||||||
LOCAL_MODEL_PATH, LOCAL_MODEL_QUANT, device = get_conf("CHATGLM_LOCAL_MODEL_PATH", "LOCAL_MODEL_QUANT", "LOCAL_MODEL_DEVICE")
|
LOCAL_MODEL_QUANT, device = get_conf("LOCAL_MODEL_QUANT", "LOCAL_MODEL_DEVICE")
|
||||||
model_path = LOCAL_MODEL_PATH
|
_model_name_ = "THUDM/chatglm3-6b"
|
||||||
|
# if LOCAL_MODEL_QUANT == "INT4": # INT4
|
||||||
|
# _model_name_ = "THUDM/chatglm3-6b-int4"
|
||||||
|
# elif LOCAL_MODEL_QUANT == "INT8": # INT8
|
||||||
|
# _model_name_ = "THUDM/chatglm3-6b-int8"
|
||||||
|
# else:
|
||||||
|
# _model_name_ = "THUDM/chatglm3-6b" # FP16
|
||||||
with ProxyNetworkActivate("Download_LLM"):
|
with ProxyNetworkActivate("Download_LLM"):
|
||||||
chatglm_tokenizer = AutoTokenizer.from_pretrained(
|
chatglm_tokenizer = AutoTokenizer.from_pretrained(
|
||||||
model_path, trust_remote_code=True
|
_model_name_, trust_remote_code=True
|
||||||
)
|
)
|
||||||
if device == "cpu":
|
if device == "cpu":
|
||||||
chatglm_model = AutoModel.from_pretrained(
|
chatglm_model = AutoModel.from_pretrained(
|
||||||
model_path,
|
_model_name_,
|
||||||
trust_remote_code=True,
|
trust_remote_code=True,
|
||||||
device="cpu",
|
device="cpu",
|
||||||
).float()
|
).float()
|
||||||
elif LOCAL_MODEL_QUANT == "INT4": # INT4
|
elif LOCAL_MODEL_QUANT == "INT4": # INT4
|
||||||
chatglm_model = AutoModel.from_pretrained(
|
chatglm_model = AutoModel.from_pretrained(
|
||||||
pretrained_model_name_or_path=model_path,
|
pretrained_model_name_or_path=_model_name_,
|
||||||
trust_remote_code=True,
|
trust_remote_code=True,
|
||||||
quantization_config=BitsAndBytesConfig(load_in_4bit=True),
|
quantization_config=BitsAndBytesConfig(load_in_4bit=True),
|
||||||
)
|
)
|
||||||
elif LOCAL_MODEL_QUANT == "INT8": # INT8
|
elif LOCAL_MODEL_QUANT == "INT8": # INT8
|
||||||
chatglm_model = AutoModel.from_pretrained(
|
chatglm_model = AutoModel.from_pretrained(
|
||||||
pretrained_model_name_or_path=model_path,
|
pretrained_model_name_or_path=_model_name_,
|
||||||
trust_remote_code=True,
|
trust_remote_code=True,
|
||||||
quantization_config=BitsAndBytesConfig(load_in_8bit=True),
|
quantization_config=BitsAndBytesConfig(load_in_8bit=True),
|
||||||
)
|
)
|
||||||
else:
|
else:
|
||||||
chatglm_model = AutoModel.from_pretrained(
|
chatglm_model = AutoModel.from_pretrained(
|
||||||
pretrained_model_name_or_path=model_path,
|
pretrained_model_name_or_path=_model_name_,
|
||||||
trust_remote_code=True,
|
trust_remote_code=True,
|
||||||
device="cuda",
|
device="cuda",
|
||||||
)
|
)
|
||||||
|
|||||||
@@ -1,81 +0,0 @@
|
|||||||
model_name = "ChatGLM4"
|
|
||||||
cmd_to_install = """
|
|
||||||
`pip install -r request_llms/requirements_chatglm4.txt`
|
|
||||||
`pip install modelscope`
|
|
||||||
`modelscope download --model ZhipuAI/glm-4-9b-chat --local_dir ./THUDM/glm-4-9b-chat`
|
|
||||||
"""
|
|
||||||
|
|
||||||
|
|
||||||
from toolbox import get_conf, ProxyNetworkActivate
|
|
||||||
from .local_llm_class import LocalLLMHandle, get_local_llm_predict_fns
|
|
||||||
|
|
||||||
|
|
||||||
# ------------------------------------------------------------------------------------------------------------------------
|
|
||||||
# 🔌💻 Local Model
|
|
||||||
# ------------------------------------------------------------------------------------------------------------------------
|
|
||||||
class GetGLM4Handle(LocalLLMHandle):
|
|
||||||
|
|
||||||
def load_model_info(self):
|
|
||||||
# 🏃♂️🏃♂️🏃♂️ 子进程执行
|
|
||||||
self.model_name = model_name
|
|
||||||
self.cmd_to_install = cmd_to_install
|
|
||||||
|
|
||||||
def load_model_and_tokenizer(self):
|
|
||||||
# 🏃♂️🏃♂️🏃♂️ 子进程执行
|
|
||||||
import torch
|
|
||||||
from transformers import AutoModel, AutoModelForCausalLM, AutoTokenizer
|
|
||||||
import os
|
|
||||||
|
|
||||||
LOCAL_MODEL_PATH, device = get_conf("CHATGLM_LOCAL_MODEL_PATH", "LOCAL_MODEL_DEVICE")
|
|
||||||
model_path = LOCAL_MODEL_PATH
|
|
||||||
chatglm_tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
|
|
||||||
chatglm_model = AutoModelForCausalLM.from_pretrained(
|
|
||||||
model_path,
|
|
||||||
torch_dtype=torch.bfloat16,
|
|
||||||
low_cpu_mem_usage=True,
|
|
||||||
trust_remote_code=True,
|
|
||||||
device=device
|
|
||||||
).eval().to(device)
|
|
||||||
self._model = chatglm_model
|
|
||||||
self._tokenizer = chatglm_tokenizer
|
|
||||||
return self._model, self._tokenizer
|
|
||||||
|
|
||||||
|
|
||||||
def llm_stream_generator(self, **kwargs):
|
|
||||||
# 🏃♂️🏃♂️🏃♂️ 子进程执行
|
|
||||||
def adaptor(kwargs):
|
|
||||||
query = kwargs["query"]
|
|
||||||
max_length = kwargs["max_length"]
|
|
||||||
top_p = kwargs["top_p"]
|
|
||||||
temperature = kwargs["temperature"]
|
|
||||||
history = kwargs["history"]
|
|
||||||
return query, max_length, top_p, temperature, history
|
|
||||||
|
|
||||||
query, max_length, top_p, temperature, history = adaptor(kwargs)
|
|
||||||
inputs = self._tokenizer.apply_chat_template([{"role": "user", "content": query}],
|
|
||||||
add_generation_prompt=True,
|
|
||||||
tokenize=True,
|
|
||||||
return_tensors="pt",
|
|
||||||
return_dict=True
|
|
||||||
).to(self._model.device)
|
|
||||||
gen_kwargs = {"max_length": max_length, "do_sample": True, "top_k": top_p}
|
|
||||||
|
|
||||||
outputs = self._model.generate(**inputs, **gen_kwargs)
|
|
||||||
outputs = outputs[:, inputs['input_ids'].shape[1]:]
|
|
||||||
response = self._tokenizer.decode(outputs[0], skip_special_tokens=True)
|
|
||||||
yield response
|
|
||||||
|
|
||||||
def try_to_import_special_deps(self, **kwargs):
|
|
||||||
# import something that will raise error if the user does not install requirement_*.txt
|
|
||||||
# 🏃♂️🏃♂️🏃♂️ 主进程执行
|
|
||||||
import importlib
|
|
||||||
|
|
||||||
# importlib.import_module('modelscope')
|
|
||||||
|
|
||||||
|
|
||||||
# ------------------------------------------------------------------------------------------------------------------------
|
|
||||||
# 🔌💻 GPT-Academic Interface
|
|
||||||
# ------------------------------------------------------------------------------------------------------------------------
|
|
||||||
predict_no_ui_long_connection, predict = get_local_llm_predict_fns(
|
|
||||||
GetGLM4Handle, model_name, history_format="chatglm3"
|
|
||||||
)
|
|
||||||
@@ -75,7 +75,7 @@ def predict_no_ui_long_connection(inputs, llm_kwargs, history=[], sys_prompt="",
|
|||||||
# make a POST request to the API endpoint, stream=False
|
# make a POST request to the API endpoint, stream=False
|
||||||
from .bridge_all import model_info
|
from .bridge_all import model_info
|
||||||
endpoint = model_info[llm_kwargs['llm_model']]['endpoint']
|
endpoint = model_info[llm_kwargs['llm_model']]['endpoint']
|
||||||
response = requests.post(endpoint, headers=headers, proxies=None,
|
response = requests.post(endpoint, headers=headers, proxies=proxies,
|
||||||
json=payload, stream=True, timeout=TIMEOUT_SECONDS); break
|
json=payload, stream=True, timeout=TIMEOUT_SECONDS); break
|
||||||
except requests.exceptions.ReadTimeout as e:
|
except requests.exceptions.ReadTimeout as e:
|
||||||
retry += 1
|
retry += 1
|
||||||
@@ -152,12 +152,10 @@ def predict(inputs, llm_kwargs, plugin_kwargs, chatbot, history=[], system_promp
|
|||||||
history.append(inputs); history.append("")
|
history.append(inputs); history.append("")
|
||||||
|
|
||||||
retry = 0
|
retry = 0
|
||||||
if proxies is not None:
|
|
||||||
logger.error("Ollama不会使用代理服务器, 忽略了proxies的设置。")
|
|
||||||
while True:
|
while True:
|
||||||
try:
|
try:
|
||||||
# make a POST request to the API endpoint, stream=True
|
# make a POST request to the API endpoint, stream=True
|
||||||
response = requests.post(endpoint, headers=headers, proxies=None,
|
response = requests.post(endpoint, headers=headers, proxies=proxies,
|
||||||
json=payload, stream=True, timeout=TIMEOUT_SECONDS);break
|
json=payload, stream=True, timeout=TIMEOUT_SECONDS);break
|
||||||
except:
|
except:
|
||||||
retry += 1
|
retry += 1
|
||||||
|
|||||||
@@ -1,7 +0,0 @@
|
|||||||
protobuf
|
|
||||||
cpm_kernels
|
|
||||||
torch>=1.10
|
|
||||||
transformers>=4.44
|
|
||||||
mdtex2html
|
|
||||||
sentencepiece
|
|
||||||
accelerate
|
|
||||||
@@ -1,46 +0,0 @@
|
|||||||
https://public.agent-matrix.com/publish/gradio-3.32.12-py3-none-any.whl
|
|
||||||
fastapi==0.110
|
|
||||||
gradio-client==0.8
|
|
||||||
pypdf2==2.12.1
|
|
||||||
httpx<=0.25.2
|
|
||||||
zhipuai==2.0.1
|
|
||||||
tiktoken>=0.3.3
|
|
||||||
requests[socks]
|
|
||||||
pydantic==2.9.2
|
|
||||||
protobuf==3.20
|
|
||||||
transformers>=4.27.1,<4.42
|
|
||||||
scipdf_parser>=0.52
|
|
||||||
spacy==3.7.4
|
|
||||||
anthropic>=0.18.1
|
|
||||||
python-markdown-math
|
|
||||||
pymdown-extensions
|
|
||||||
websocket-client
|
|
||||||
beautifulsoup4
|
|
||||||
prompt_toolkit
|
|
||||||
latex2mathml
|
|
||||||
python-docx
|
|
||||||
mdtex2html
|
|
||||||
dashscope
|
|
||||||
pyautogen
|
|
||||||
colorama
|
|
||||||
Markdown
|
|
||||||
pygments
|
|
||||||
edge-tts>=7.0.0
|
|
||||||
pymupdf
|
|
||||||
openai
|
|
||||||
rjsmin
|
|
||||||
loguru
|
|
||||||
arxiv
|
|
||||||
numpy
|
|
||||||
rich
|
|
||||||
|
|
||||||
|
|
||||||
llama-index-core==0.10.68
|
|
||||||
llama-index-legacy==0.9.48
|
|
||||||
llama-index-readers-file==0.1.33
|
|
||||||
llama-index-readers-llama-parse==0.1.6
|
|
||||||
llama-index-embeddings-azure-openai==0.1.10
|
|
||||||
llama-index-embeddings-openai==0.1.10
|
|
||||||
llama-parse==0.4.9
|
|
||||||
mdit-py-plugins>=0.3.3
|
|
||||||
linkify-it-py==2.0.3
|
|
||||||
@@ -77,28 +77,16 @@ def make_history_cache():
|
|||||||
# 定义 后端state（history）、前端（history_cache）、后端setter（history_cache_update）三兄弟
|
# 定义 后端state（history）、前端（history_cache）、后端setter（history_cache_update）三兄弟
|
||||||
import gradio as gr
|
import gradio as gr
|
||||||
# 定义history的后端state
|
# 定义history的后端state
|
||||||
# history = gr.State([])
|
history = gr.State([])
|
||||||
history = gr.Textbox(visible=False, elem_id="history-ng")
|
# 定义history的一个孪生的前端存储区（隐藏）
|
||||||
# # 定义history的一个孪生的前端存储区（隐藏）
|
history_cache = gr.Textbox(visible=False, elem_id="history_cache")
|
||||||
# history_cache = gr.Textbox(visible=False, elem_id="history_cache")
|
# 定义history_cache->history的更新方法（隐藏）。在触发这个按钮时，会先执行js代码更新history_cache，然后再执行python代码更新history
|
||||||
# # 定义history_cache->history的更新方法（隐藏）。在触发这个按钮时，会先执行js代码更新history_cache，然后再执行python代码更新history
|
def process_history_cache(history_cache):
|
||||||
# def process_history_cache(history_cache):
|
return json.loads(history_cache)
|
||||||
# return json.loads(history_cache)
|
# 另一种更简单的setter方法
|
||||||
# # 另一种更简单的setter方法
|
history_cache_update = gr.Button("", elem_id="elem_update_history", visible=False).click(
|
||||||
# history_cache_update = gr.Button("", elem_id="elem_update_history", visible=False).click(
|
process_history_cache, inputs=[history_cache], outputs=[history])
|
||||||
# process_history_cache, inputs=[history_cache], outputs=[history])
|
return history, history_cache, history_cache_update
|
||||||
# # save history to history_cache
|
|
||||||
# def process_history_cache(history_cache):
|
|
||||||
# return json.dumps(history_cache)
|
|
||||||
# # 定义history->history_cache的更新方法（隐藏）
|
|
||||||
# def sync_history_cache(history):
|
|
||||||
# print("sync_history_cache", history)
|
|
||||||
# return json.dumps(history)
|
|
||||||
# # history.change(sync_history_cache, inputs=[history], outputs=[history_cache])
|
|
||||||
|
|
||||||
# # history_cache_sync = gr.Button("", elem_id="elem_sync_history", visible=False).click(
|
|
||||||
# # lambda history: (json.dumps(history)), inputs=[history_cache], outputs=[history])
|
|
||||||
return history, None, None
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -1,10 +0,0 @@
|
|||||||
def validate_path():
|
|
||||||
import os, sys
|
|
||||||
|
|
||||||
os.path.dirname(__file__)
|
|
||||||
root_dir_assume = os.path.abspath(os.path.dirname(__file__) + "/..")
|
|
||||||
os.chdir(root_dir_assume)
|
|
||||||
sys.path.append(root_dir_assume)
|
|
||||||
|
|
||||||
|
|
||||||
validate_path() # validate path so you can run from base directory
|
|
||||||
@@ -1,12 +0,0 @@
|
|||||||
"""
|
|
||||||
对项目中的各个插件进行测试。运行方法：直接运行 python tests/test_plugins.py
|
|
||||||
"""
|
|
||||||
|
|
||||||
import init_test
|
|
||||||
import os, sys
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
from test_utils import plugin_test
|
|
||||||
|
|
||||||
plugin_test(plugin='crazy_functions.数学动画生成manim->动画生成', main_input="A point moving along function culve y=sin(x), starting from x=0 and stop at x=4*\pi.")
|
|
||||||
@@ -1,15 +0,0 @@
|
|||||||
"""
|
|
||||||
对项目中的各个插件进行测试。运行方法：直接运行 python tests/test_plugins.py
|
|
||||||
"""
|
|
||||||
|
|
||||||
import init_test
|
|
||||||
import os, sys
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
from experimental_mods.get_bilibili_resource import download_bilibili
|
|
||||||
download_bilibili("BV1LSSHYXEtv", only_audio=True, user_name="test")
|
|
||||||
|
|
||||||
# if __name__ == "__main__":
|
|
||||||
# from test_utils import plugin_test
|
|
||||||
|
|
||||||
# plugin_test(plugin='crazy_functions.VideoResource_GPT->视频任务', main_input="帮我找到《天文馆的猫》，歌手泠鸢")
|
|
||||||
@@ -1,7 +0,0 @@
|
|||||||
import init_test
|
|
||||||
|
|
||||||
from crazy_functions.pdf_fns.parse_pdf_via_doc2x import 解析PDF_DOC2X_转Latex
|
|
||||||
|
|
||||||
# 解析PDF_DOC2X_转Latex("gpt_log/arxiv_cache_old/2410.10819/workfolder/merge.pdf")
|
|
||||||
# 解析PDF_DOC2X_转Latex("gpt_log/arxiv_cache_ooo/2410.07095/workfolder/merge.pdf")
|
|
||||||
解析PDF_DOC2X_转Latex("2410.11190v2.pdf")
|
|
||||||
@@ -1,586 +0,0 @@
|
|||||||
def validate_path():
|
|
||||||
import os, sys
|
|
||||||
os.path.dirname(__file__)
|
|
||||||
root_dir_assume = os.path.abspath(os.path.dirname(__file__) + "/..")
|
|
||||||
os.chdir(root_dir_assume)
|
|
||||||
sys.path.append(root_dir_assume)
|
|
||||||
|
|
||||||
validate_path() # validate path so you can run from base directory
|
|
||||||
|
|
||||||
# # """
|
|
||||||
# # Test 1
|
|
||||||
# # """
|
|
||||||
|
|
||||||
# # from request_llms.embed_models.openai_embed import OpenAiEmbeddingModel
|
|
||||||
# # from shared_utils.connect_void_terminal import get_chat_default_kwargs
|
|
||||||
# # oaiem = OpenAiEmbeddingModel()
|
|
||||||
|
|
||||||
# # chat_kwargs = get_chat_default_kwargs()
|
|
||||||
# # llm_kwargs = chat_kwargs['llm_kwargs']
|
|
||||||
# # llm_kwargs.update({
|
|
||||||
# # 'llm_model': "text-embedding-3-small"
|
|
||||||
# # })
|
|
||||||
|
|
||||||
# # res = oaiem.compute_embedding("你好", llm_kwargs)
|
|
||||||
# # print(res)
|
|
||||||
|
|
||||||
# """
|
|
||||||
# Test 2
|
|
||||||
# """
|
|
||||||
|
|
||||||
# from request_llms.embed_models.openai_embed import OpenAiEmbeddingModel
|
|
||||||
from shared_utils.connect_void_terminal import get_chat_default_kwargs
|
|
||||||
# from llama_index.core import VectorStoreIndex, SimpleDirectoryReader
|
|
||||||
# from crazy_functions.rag_fns.vector_store_index import GptacVectorStoreIndex
|
|
||||||
# from llama_index.core.ingestion import run_transformations
|
|
||||||
# from llama_index.core import PromptTemplate
|
|
||||||
# from llama_index.core.response_synthesizers import TreeSummarize
|
|
||||||
|
|
||||||
# # NOTE: we add an extra tone_name variable here
|
|
||||||
# DEFAULT_QUESTION_GENERATION_PROMPT = """\
|
|
||||||
# Context information is below.
|
|
||||||
# ---------------------
|
|
||||||
# {context_str}
|
|
||||||
# ---------------------
|
|
||||||
# Given the context information and not prior knowledge.
|
|
||||||
# generate only questions based on the below query.
|
|
||||||
# {query_str}
|
|
||||||
# """
|
|
||||||
|
|
||||||
|
|
||||||
chat_kwargs = get_chat_default_kwargs()
|
|
||||||
llm_kwargs = chat_kwargs['llm_kwargs']
|
|
||||||
llm_kwargs.update({
|
|
||||||
'llm_model': "text-embedding-3-small",
|
|
||||||
'embed_model': "text-embedding-3-small"
|
|
||||||
})
|
|
||||||
# embed_model = OpenAiEmbeddingModel(llm_kwargs)
|
|
||||||
|
|
||||||
# ## dir
|
|
||||||
# documents = SimpleDirectoryReader("private_upload/rag_test/").load_data()
|
|
||||||
|
|
||||||
# ## single files
|
|
||||||
# # from llama_index.core import Document
|
|
||||||
# # text_list = [text1, text2, ...]
|
|
||||||
# # documents = [Document(text=t) for t in text_list]
|
|
||||||
# vsi = GptacVectorStoreIndex.default_vector_store(embed_model=embed_model)
|
|
||||||
# documents_nodes = run_transformations(
|
|
||||||
# documents, # type: ignore
|
|
||||||
# vsi._transformations,
|
|
||||||
# show_progress=True
|
|
||||||
# )
|
|
||||||
# index = vsi.insert_nodes(documents_nodes)
|
|
||||||
# retriever = vsi.as_retriever()
|
|
||||||
|
|
||||||
# query = "what is core_functional.py"
|
|
||||||
|
|
||||||
# res = retriever.retrieve(query)
|
|
||||||
# context_str = '\n'.join([r.text for r in res])
|
|
||||||
# query_str = query
|
|
||||||
# query = DEFAULT_QUESTION_GENERATION_PROMPT.format(context_str=context_str, query_str=query_str)
|
|
||||||
# print(res)
|
|
||||||
# print(res)
|
|
||||||
|
|
||||||
|
|
||||||
# # response = query_engine.query("Some question about the data should go here")
|
|
||||||
# # print(response)
|
|
||||||
|
|
||||||
from crazy_functions.rag_fns.llama_index_worker import LlamaIndexRagWorker
|
|
||||||
rag_worker = LlamaIndexRagWorker('good-man-user', llm_kwargs, checkpoint_dir='./longlong_vector_store')
|
|
||||||
|
|
||||||
rag_worker.add_text_to_vector_store("""
|
|
||||||
熊童子（Cotyledon tomentosa）是景天科，银波锦属的多年生肉质草本植物，植株多分枝，茎绿色，肉质叶肥厚，交互对生，卵圆形，绿色，密生白色短毛。叶端具红色爪样齿，二歧聚伞花序，小花黄色，花期7-9月。
|
|
||||||
该种原产于南非开普省。喜温暖干燥，阳光充足，通风良好的环境。夏季温度过高会休眠。忌寒冷和过分潮湿。繁殖方法有扦插。
|
|
||||||
该种叶形叶色较美，花朵玲珑小巧，叶片形似小熊的脚掌，形态奇特，十分可爱，观赏价值很高。
|
|
||||||
物种索引：IN4679748
|
|
||||||
""")
|
|
||||||
|
|
||||||
rag_worker.add_text_to_vector_store("""
|
|
||||||
碧光环是番杏科碧光玉属 [4]多年生肉质草本植物。 [5]碧光环叶表面有半透明的颗粒感，晶莹剔透；两片圆柱形的叶子，在生长初期像兔耳，非常可爱，长大后叶子会慢慢变长变粗，缺水时容易耷拉下来；具枝干，易群生。
|
|
||||||
碧光环原产于南非。碧光环喜温暖和散射光充足的环境，较耐寒，忌强光暴晒，夏季高温休眠明显。 [6]碧光环的繁殖方式有扦插和播种。 [7]
|
|
||||||
碧光环小巧饱满、圆滚滚的样子很可爱，长得好像长耳朵小兔，萌萌的样子让人爱不释手，而且养起来也不难，极具观赏价值。 [8]
|
|
||||||
物种索引：IN985654
|
|
||||||
""")
|
|
||||||
|
|
||||||
rag_worker.add_text_to_vector_store("""
|
|
||||||
福娘为景天科银波锦属的肉质草本植物。对生的叶片呈短棒状，叶色灰绿，表覆白粉，叶缘外围镶着紫红色，叶片外形多有变化有短圆形、厚厚的方形等不同叶形； [5]花期夏秋。 [6]
|
|
||||||
福娘原产于非洲西南部的纳米比亚，现世界多地均有栽培。性喜欢凉爽通风、日照充足的环境，较喜光照，喜肥，生长适温为15-25℃，冬季温度不低于5℃，生长期要见干见湿。 [7]在通风透气、排水良好的土壤上生长良好，一般可用泥炭土、蛭石和珍珠岩的混合土。繁殖方式一般为扦插繁殖，多用枝插，叶插的繁殖成功率不高。 [8]
|
|
||||||
因福娘的叶形叶色较美，所以具有一定的观赏价值，可盆栽放置于电视、电脑旁，吸收辐射，亦可栽植于室内以吸收甲醛等物质，净化空气。 [9]
|
|
||||||
物种索引：IN772
|
|
||||||
""")
|
|
||||||
|
|
||||||
rag_worker.add_text_to_vector_store("""
|
|
||||||
石莲（ Sinocrassula indica (Decne.) A. Berger）是景天科石莲属 [8]的二年生草本植物。基生叶莲座状，匙状长圆形；茎生叶互生，宽倒披针状线形或近倒卵形；花序圆锥状或近伞房状，萼片呈宽三角形，花瓣呈红色，披针形或卵形，雄蕊呈正方形；蓇葖果的喙反曲；种子平滑；花期9月；果期10月 [9]。锯叶石莲为石莲的变种，与原变种的不同处为叶上部有渐尖的锯齿。茎和花无毛，叶被毛 [10]。因叶子有棱有角，又似玉石，故而得名“石莲” [11]。
|
|
||||||
物种索引：IN455674
|
|
||||||
""")
|
|
||||||
|
|
||||||
rag_worker.add_text_to_vector_store("""
|
|
||||||
虹之玉锦（Sedum × rubrotinctum 'Aurora'） [1]是景天科景天属的多肉植物，为虹之玉的锦化品种。虹之玉锦与虹之玉的叶片大小没有特别大的变化，但颜色会稍有不同，虹之玉锦一般会有粉红色、中绿色等 [2]。生长速度较虹之玉慢很多 [3]。
|
|
||||||
物种索引：IN88
|
|
||||||
""")
|
|
||||||
|
|
||||||
rag_worker.add_text_to_vector_store("""
|
|
||||||
一个幽灵，共产主义的幽灵，在欧洲游荡。为了对这个幽灵进行神圣的围剿，旧欧洲的一切势力，教皇和沙皇、梅特涅和基佐、法国的激进派和德国的警察，都联合起来了。
|
|
||||||
|
|
||||||
有哪一个反对党不被它的当政的敌人骂为共产党呢？又有哪一个反对党不拿共产主义这个罪名去回敬更进步的反对党人和自己的反动敌人呢？
|
|
||||||
|
|
||||||
从这一事实中可以得出两个结论：
|
|
||||||
|
|
||||||
共产主义已经被欧洲的一切势力公认为一种势力；
|
|
||||||
|
|
||||||
现在是共产党人向全世界公开说明自己的观点、自己的目的、自己的意图并且拿党自己的宣言来反驳关于共产主义幽灵的神话的时候了。
|
|
||||||
|
|
||||||
为了这个目的，各国共产党人集会于伦敦，拟定了如下的宣言，用英文、法文、德文、意大利文、弗拉芒文和丹麦文公布于世。
|
|
||||||
|
|
||||||
一、资产者和无产者
|
|
||||||
|
|
||||||
至今一切社会的历史都是阶级斗争的历史。
|
|
||||||
|
|
||||||
自由民和奴隶、贵族和平民、领主和农奴、行会师傅和帮工，一句话，压迫者和被压迫者，始终处于相互对立的地位，进行不断的、有时隐蔽有时公开的斗争，而每一次斗争的结局都是整个社会受到革命改造或者斗争的各阶级同归于尽。
|
|
||||||
|
|
||||||
在过去的各个历史时代，我们几乎到处都可以看到社会完全划分为各个不同的等级，看到社会地位分成多种多样的层次。在古罗马，有贵族、骑士、平民、奴隶，在中世纪，有封建主、臣仆、行会师傅、帮工、农奴，而且几乎在每一个阶级内部又有一些特殊的阶层。
|
|
||||||
|
|
||||||
从封建社会的灭亡中产生出来的现代资产阶级社会并没有消灭阶级对立。它只是用新的阶级、新的压迫条件、新的斗争形式代替了旧的。
|
|
||||||
|
|
||||||
但是，我们的时代，资产阶级时代，却有一个特点：它使阶级对立简单化了。整个社会日益分裂为两大敌对的阵营，分裂为两大相互直接对立的阶级：资产阶级和无产阶级。
|
|
||||||
|
|
||||||
从中世纪的农奴中产生了初期城市的城关市民；从这个市民等级中发展出最初的资产阶级分子。
|
|
||||||
|
|
||||||
美洲的发现、绕过非洲的航行，给新兴的资产阶级开辟了新天地。东印度和中国的市场、美洲的殖民化、对殖民地的贸易、交换手段和一般商品的增加，使商业、航海业和工业空前高涨，因而使正在崩溃的封建社会内部的革命因素迅速发展。
|
|
||||||
|
|
||||||
以前那种封建的或行会的工业经营方式已经不能满足随着新市场的出现而增加的需求了。工场手工业代替了这种经营方式。行会师傅被工业的中间等级排挤掉了；各种行业组织之间的分工随着各个作坊内部的分工的出现而消失了。
|
|
||||||
|
|
||||||
但是，市场总是在扩大，需求总是在增加。甚至工场手工业也不再能满足需要了。于是，蒸汽和机器引起了工业生产的革命。现代大工业代替了工场手工业；工业中的百万富翁，一支一支产业大军的首领，现代资产者，代替了工业的中间等级。
|
|
||||||
|
|
||||||
大工业建立了由美洲的发现所准备好的世界市场。世界市场使商业、航海业和陆路交通得到了巨大的发展。这种发展又反过来促进了工业的扩展。同时，随着工业、商业、航海业和铁路的扩展，资产阶级也在同一程度上得到发展，增加自己的资本，把中世纪遗留下来的一切阶级排挤到后面去。
|
|
||||||
|
|
||||||
由此可见，现代资产阶级本身是一个长期发展过程的产物，是生产方式和交换方式的一系列变革的产物。
|
|
||||||
|
|
||||||
资产阶级的这种发展的每一个阶段，都伴随着相应的政治上的进展。它在封建主统治下是被压迫的等级，在公社里是武装的和自治的团体，在一些地方组成独立的城市共和国，在另一些地方组成君主国中的纳税的第三等级；后来，在工场手工业时期，它是等级君主国或专制君主国中同贵族抗衡的势力，而且是大君主国的主要基础；最后，从大工业和世界市场建立的时候起，它在现代的代议制国家里夺得了独占的政治统治。现代的国家政权不过是管理整个资产阶级的共同事务的委员会罢了。
|
|
||||||
|
|
||||||
资产阶级在历史上曾经起过非常革命的作用。
|
|
||||||
|
|
||||||
资产阶级在它已经取得了统治的地方把一切封建的、宗法的和田园般的关系都破坏了。它无情地斩断了把人们束缚于天然尊长的形形色色的封建羁绊，它使人和人之间除了赤裸裸的利害关系，除了冷酷无情的“现金交易”，就再也没有任何别的联系了。它把宗教虔诚、骑士热忱、小市民伤感这些情感的神圣发作，淹没在利己主义打算的冰水之中。它把人的尊严变成了交换价值，用一种没有良心的贸易自由代替了无数特许的和自力挣得的自由。总而言之，它用公开的、无耻的、直接的、露骨的剥削代替了由宗教幻想和政治幻想掩盖着的剥削。
|
|
||||||
|
|
||||||
资产阶级抹去了一切向来受人尊崇和令人敬畏的职业的神圣光环。它把医生、律师、教士、诗人和学者变成了它出钱招雇的雇佣劳动者。
|
|
||||||
|
|
||||||
资产阶级撕下了罩在家庭关系上的温情脉脉的面纱，把这种关系变成了纯粹的金钱关系。
|
|
||||||
|
|
||||||
资产阶级揭示了，在中世纪深受反动派称许的那种人力的野蛮使用，是以极端怠惰作为相应补充的。它第一个证明了，人的活动能够取得什么样的成就。它创造了完全不同于埃及金字塔、罗马水道和哥特式教堂的奇迹；它完成了完全不同于民族大迁徙和十字军征讨的远征。
|
|
||||||
|
|
||||||
资产阶级除非对生产工具，从而对生产关系，从而对全部社会关系不断地进行革命，否则就不能生存下去。反之，原封不动地保持旧的生产方式，却是过去的一切工业阶级生存的首要条件。生产的不断变革，一切社会状况不停的动荡，永远的不安定和变动，这就是资产阶级时代不同于过去一切时代的地方。一切固定的僵化的关系以及与之相适应的素被尊崇的观念和见解都被消除了，一切新形成的关系等不到固定下来就陈旧了。一切等级的和固定的东西都烟消云散了，一切神圣的东西都被亵渎了。人们终于不得不用冷静的眼光来看他们的生活地位、他们的相互关系。
|
|
||||||
|
|
||||||
不断扩大产品销路的需要，驱使资产阶级奔走于全球各地。它必须到处落户，到处开发，到处建立联系。
|
|
||||||
|
|
||||||
资产阶级，由于开拓了世界市场，使一切国家的生产和消费都成为世界性的了。使反动派大为惋惜的是，资产阶级挖掉了工业脚下的民族基础。古老的民族工业被消灭了，并且每天都还在被消灭。它们被新的工业排挤掉了，新的工业的建立已经成为一切文明民族的生命攸关的问题；这些工业所加工的，已经不是本地的原料，而是来自极其遥远的地区的原料；它们的产品不仅供本国消费，而且同时供世界各地消费。旧的、靠本国产品来满足的需要，被新的、要靠极其遥远的国家和地带的产品来满足的需要所代替了。过去那种地方的和民族的自给自足和闭关自守状态，被各民族的各方面的互相往来和各方面的互相依赖所代替了。物质的生产是如此，精神的生产也是如此。各民族的精神产品成了公共的财产。民族的片面性和局限性日益成为不可能，于是由许多种民族的和地方的文学形成了一种世界的文学。
|
|
||||||
|
|
||||||
资产阶级，由于一切生产工具的迅速改进，由于交通的极其便利，把一切民族甚至最野蛮的民族都卷到文明中来了。它的商品的低廉价格，是它用来摧毁一切万里长城、征服野蛮人最顽强的仇外心理的重炮。它迫使一切民族——如果它们不想灭亡的话——采用资产阶级的生产方式；它迫使它们在自己那里推行所谓的文明，即变成资产者。一句话，它按照自己的面貌为自己创造出一个世界。
|
|
||||||
|
|
||||||
资产阶级使农村屈服于城市的统治。它创立了巨大的城市，使城市人口比农村人口大大增加起来，因而使很大一部分居民脱离了农村生活的愚昧状态。正像它使农村从属于城市一样，它使未开化和半开化的国家从属于文明的国家，使农民的民族从属于资产阶级的民族，使东方从属于西方。
|
|
||||||
|
|
||||||
资产阶级日甚一日地消灭生产资料、财产和人口的分散状态。它使人口密集起来，使生产资料集中起来，使财产聚集在少数人的手里。由此必然产生的结果就是政治的集中。各自独立的、几乎只有同盟关系的、各有不同利益、不同法律、不同政府、不同关税的各个地区，现在已经结合为一个拥有统一的政府、统一的法律、统一的民族阶级利益和统一的关税的统一的民族。
|
|
||||||
|
|
||||||
资产阶级在它的不到一百年的阶级统治中所创造的生产力，比过去一切世代创造的全部生产力还要多，还要大。自然力的征服，机器的采用，化学在工业和农业中的应用，轮船的行驶，铁路的通行，电报的使用，整个整个大陆的开垦，河川的通航，仿佛用法术从地下呼唤出来的大量人口，——过去哪一个世纪料想到在社会劳动里蕴藏有这样的生产力呢？
|
|
||||||
|
|
||||||
由此可见，资产阶级赖以形成的生产资料和交换手段，是在封建社会里造成的。在这些生产资料和交换手段发展的一定阶段上，封建社会的生产和交换在其中进行的关系，封建的农业和工场手工业组织，一句话，封建的所有制关系，就不再适应已经发展的生产力了。这种关系已经在阻碍生产而不是促进生产了。它变成了束缚生产的桎梏。它必须被炸毁，它已经被炸毁了。
|
|
||||||
|
|
||||||
起而代之的是自由竞争以及与自由竞争相适应的社会制度和政治制度、资产阶级的经济统治和政治统治。
|
|
||||||
|
|
||||||
现在，我们眼前又进行着类似的运动。资产阶级的生产关系和交换关系，资产阶级的所有制关系，这个曾经仿佛用法术创造了如此庞大的生产资料和交换手段的现代资产阶级社会，现在像一个魔法师一样不能再支配自己用法术呼唤出来的魔鬼了。几十年来的工业和商业的历史，只不过是现代生产力反抗现代生产关系、反抗作为资产阶级及其统治的存在条件的所有制关系的历史。只要指出在周期性的重复中越来越危及整个资产阶级社会生存的商业危机就够了。在商业危机期间，总是不仅有很大一部分制成的产品被毁灭掉，而且有很大一部分已经造成的生产力被毁灭掉。在危机期间，发生一种在过去一切时代看来都好像是荒唐现象的社会瘟疫，即生产过剩的瘟疫。社会突然发现自己回到了一时的野蛮状态；仿佛是一次饥荒、一场普遍的毁灭性战争，使社会失去了全部生活资料；仿佛是工业和商业全被毁灭了，——这是什么缘故呢？因为社会上文明过度，生活资料太多，工业和商业太发达。社会所拥有的生产力已经不能再促进资产阶级文明和资产阶级所有制关系的发展；相反，生产力已经强大到这种关系所不能适应的地步，它已经受到这种关系的阻碍；而它一着手克服这种障碍，就使整个资产阶级社会陷入混乱，就使资产阶级所有制的存在受到威胁。资产阶级的关系已经太狭窄了，再容纳不了它本身所造成的财富了。——资产阶级用什么办法来克服这种危机呢？一方面不得不消灭大量生产力，另一方面夺取新的市场，更加彻底地利用旧的市场。这究竟是怎样的一种办法呢？这不过是资产阶级准备更全面更猛烈的危机的办法，不过是使防止危机的手段越来越少的办法。
|
|
||||||
|
|
||||||
资产阶级用来推翻封建制度的武器，现在却对准资产阶级自己了。
|
|
||||||
|
|
||||||
但是，资产阶级不仅锻造了置自身于死地的武器；它还产生了将要运用这种武器的人——现代的工人，即无产者。
|
|
||||||
|
|
||||||
随着资产阶级即资本的发展，无产阶级即现代工人阶级也在同一程度上得到发展；现代的工人只有当他们找到工作的时候才能生存，而且只有当他们的劳动增殖资本的时候才能找到工作。这些不得不把自己零星出卖的工人，像其他任何货物一样，也是一种商品，所以他们同样地受到竞争的一切变化、市场的一切波动的影响。
|
|
||||||
|
|
||||||
由于推广机器和分工，无产者的劳动已经失去了任何独立的性质，因而对工人也失去了任何吸引力。工人变成了机器的单纯的附属品，要求他做的只是极其简单、极其单调和极容易学会的操作。因此，花在工人身上的费用，几乎只限于维持工人生活和延续工人后代所必需的生活资料。但是，商品的价格，从而劳动的价格，是同它的生产费用相等的。因此，劳动越使人感到厌恶，工资也就越少。不仅如此，机器越推广，分工越细致，劳动量出就越增加，这或者是由于工作时间的延长，或者是由于在一定时间内所要求的劳动的增加，机器运转的加速，等等。
|
|
||||||
|
|
||||||
现代工业已经把家长式的师傅的小作坊变成了工业资本家的大工厂。挤在工厂里的工人群众就像士兵一样被组织起来。他们是产业军的普通士兵，受着各级军士和军官的层层监视。他们不仅仅是资产阶级的、资产阶级国家的奴隶，他们每日每时都受机器、受监工、首先是受各个经营工厂的资产者本人的奴役。这种专制制度越是公开地把营利宣布为自己的最终目的，它就越是可鄙、可恨和可恶。
|
|
||||||
|
|
||||||
手的操作所要求的技巧和气力越少，换句话说，现代工业越发达，男工也就越受到女工和童工的排挤。对工人阶级来说，性别和年龄的差别再没有什么社会意义了。他们都只是劳动工具，不过因为年龄和性别的不同而需要不同的费用罢了。
|
|
||||||
|
|
||||||
当厂主对工人的剥削告一段落，工人领到了用现钱支付的工资的时候，马上就有资产阶级中的另一部分人——房东、小店主、当铺老板等等向他们扑来。
|
|
||||||
|
|
||||||
以前的中间等级的下层，即小工业家、小商人和小食利者，手工业者和农民——所有这些阶级都降落到无产阶级的队伍里来了，有的是因为他们的小资本不足以经营大工业，经不起较大的资本家的竞争；有的是因为他们的手艺已经被新的生产方法弄得不值钱了。无产阶级就是这样从居民的所有阶级中得到补充的。
|
|
||||||
|
|
||||||
无产阶级经历了各个不同的发展阶段。它反对资产阶级的斗争是和它的存在同时开始的。
|
|
||||||
|
|
||||||
最初是单个的工人，然后是某一工厂的工人，然后是某一地方的某一劳动部门的工人，同直接剥削他们的单个资产者作斗争。他们不仅仅攻击资产阶级的生产关系，而且攻击生产工具本身；他们毁坏那些来竞争的外国商品，捣毁机器，烧毁工厂，力图恢复已经失去的中世纪工人的地位。
|
|
||||||
|
|
||||||
在这个阶段上，工人是分散在全国各地并为竞争所分裂的群众。工人的大规模集结，还不是他们自己联合的结果，而是资产阶级联合的结果，当时资产阶级为了达到自己的政治目的必须而且暂时还能够把整个无产阶级发动起来。因此，在这个阶段上，无产者不是同自己的敌人作斗争，而是同自己的敌人的敌人作斗争，即同专制君主制的残余、地主、非工业资产者和小资产者作斗争。因此，整个历史运动都集中在资产阶级手里；在这种条件下取得的每一个胜利都是资产阶级的胜利。
|
|
||||||
|
|
||||||
但是，随着工业的发展，无产阶级不仅人数增加了，而且它结合成更大的集体，它的力量日益增长，它越来越感觉到自己的力量。机器使劳动的差别越来越小，使工资几乎到处都降到同样低的水平，因而无产阶级内部的利益、生活状况也越来越趋于一致。资产者彼此间日益加剧的竞争以及由此引起的商业危机，使工人的工资越来越不稳定；机器的日益迅速的和继续不断的改良，使工人的整个生活地位越来越没有保障；单个工人和单个资产者之间的冲突越来越具有两个阶级的冲突的性质。工人开始成立反对资产者的同盟；他们联合起来保卫自己的工资。他们甚至建立了经常性的团体，以便为可能发生的反抗准备食品。有些地方，斗争爆发为起义。
|
|
||||||
|
|
||||||
工人有时也得到胜利，但这种胜利只是暂时的。他们斗争的真正成果并不是直接取得的成功，而是工人的越来越扩大的联合。这种联合由于大工业所造成的日益发达的交通工具而得到发展，这种交通工具把各地的工人彼此联系起来。只要有了这种联系，就能把许多性质相同的地方性的斗争汇合成全国性的斗争，汇合成阶级斗争。而一切阶级斗争都是政治斗争。中世纪的市民靠乡间小道需要几百年才能达到的联合，现代的无产者利用铁路只要几年就可以达到了。
|
|
||||||
|
|
||||||
无产者组织成为阶级，从而组织成为政党这件事，不断地由于工人的自相竞争而受到破坏。但是，这种组织总是重新产生，并且一次比一次更强大，更坚固，更有力。它利用资产阶级内部的分裂，迫使他们用法律形式承认工人的个别利益。英国的十小时工作日法案就是一个例子。
|
|
||||||
|
|
||||||
旧社会内部的所有冲突在许多方面都促进了无产阶级的发展。资产阶级处于不断的斗争中：最初反对贵族；后来反对同工业进步有利害冲突的那部分资产阶级；经常反对一切外国的资产阶级。在这一切斗争中，资产阶级都不得不向无产阶级呼吁，要求无产阶级援助，这样就把无产阶级卷进了政治运动。于是，资产阶级自己就把自己的教育因素即反对自身的武器给予了无产阶级。
|
|
||||||
|
|
||||||
其次，我们已经看到，工业的进步把统治阶级的整批成员抛到无产阶级队伍里去，或者至少也使他们的生活条件受到威胁。他们也给无产阶级带来了大量的教育因素。
|
|
||||||
|
|
||||||
最后，在阶级斗争接近决战的时期，统治阶级内部的、整个旧社会内部的瓦解过程，就达到非常强烈、非常尖锐的程度，甚至使得统治阶级中的一小部分人脱离统治阶级而归附于革命的阶级，即掌握着未来的阶级。所以，正像过去贵族中有一部分人转到资产阶级方面一样，现在资产阶级中也有一部分人，特别是已经提高到从理论上认识整个历史运动这一水平的一部分资产阶级思想家，转到无产阶级方面来了。
|
|
||||||
|
|
||||||
在当前同资产阶级对立的一切阶级中，只有无产阶级是真正革命的阶级。其余的阶级都随着大工业的发展而日趋没落和灭亡，无产阶级却是大工业本身的产物。
|
|
||||||
|
|
||||||
中间等级，即小工业家、小商人、手工业者、农民，他们同资产阶级作斗争，都是为了维护他们这种中间等级的生存，以免于灭亡。所以，他们不是革命的，而是保守的。不仅如此，他们甚至是反动的，因为他们力图使历史的车轮倒转。如果说他们是革命的，那是鉴于他们行将转入无产阶级的队伍，这样，他们就不是维护他们目前的利益，而是维护他们将来的利益，他们就离开自己原来的立场，而站到无产阶级的立场上来。
|
|
||||||
|
|
||||||
流氓无产阶级是旧社会最下层中消极的腐化的部分，他们在一些地方也被无产阶级革命卷到运动里来，但是，由于他们的整个生活状况，他们更甘心于被人收买，去干反动的勾当。
|
|
||||||
|
|
||||||
在无产阶级的生活条件中，旧社会的生活条件已经被消灭了。无产者是没有财产的；他们和妻子儿女的关系同资产阶级的家庭关系再没有任何共同之处了；现代的工业劳动，现代的资本压迫，无论在英国或法国，无论在美国或德国，都有是一样的，都使无产者失去了任何民族性。法律、道德、宗教在他们看来全都是资产阶级偏见，隐藏在这些偏见后面的全都是资产阶级利益。
|
|
||||||
|
|
||||||
过去一切阶级在争得统治之后，总是使整个社会服从于它们发财致富的条件，企图以此来巩固它们已获得的生活地位。无产者只有废除自己的现存的占有方式，从而废除全部现存的占有方式，才能取得社会生产力。无产者没有什么自己的东西必须加以保护，他们必须摧毁至今保护和保障私有财产的一切。
|
|
||||||
|
|
||||||
过去的一切运动都是少数人的或者为少数人谋利益的运动。无产阶级的运动是绝大多数人的、为绝大多数人谋利益的独立的运动。无产阶级，现今社会的最下层，如果不炸毁构成官方社会的整个上层，就不能抬起头来，挺起胸来。
|
|
||||||
|
|
||||||
如果不就内容而就形式来说，无产阶级反对资产阶级的斗争首先是一国范围内的斗争。每一个国家的无产阶级当然首先应该打倒本国的资产阶级。
|
|
||||||
|
|
||||||
在叙述无产阶级发展的最一般的阶段的时候，我们循序探讨了现存社会内部或多或少隐蔽着的国内战争，直到这个战争爆发为公开的革命，无产阶级用暴力推翻资产阶级而建立自己的统治。
|
|
||||||
|
|
||||||
我们已经看到，至今的一切社会都是建立在压迫阶级和被压迫阶级的对立之上的。但是，为了有可能压迫一个阶级，就必须保证这个阶级至少有能够勉强维持它的奴隶般的生存的条件。农奴曾经在农奴制度下挣扎到公社成员的地位，小资产者曾经在封建专制制度的束缚下挣扎到资产者的地位。现代的工人却相反，他们并不是随着工业的进步而上升，而是越来越降到本阶级的生存条件以下。工人变成赤贫者，贫困比人口和财富增长得还要快。由此可以明显地看出，资产阶级再不能做社会的统治阶级了，再不能把自己阶级的生存条件当作支配一切的规律强加于社会了。资产阶级不能统治下去了，因为它甚至不能保证自己的奴隶维持奴隶的生活，因为它不得不让自己的奴隶落到不能养活它反而要它来养活的地步。社会再不能在它统治下生存下去了，就是说，它的生存不再同社会相容了。
|
|
||||||
|
|
||||||
资产阶级生存和统治的根本条件，是财富在私人手里的积累，是资本的形成和增殖；资本的条件是雇佣劳动。雇佣劳动完全是建立在工人的自相竞争之上的。资产阶级无意中造成而又无力抵抗的工业进步，使工人通过结社而达到的革命联合代替了他们由于竞争而造成的分散状态。于是，随着大工业的发展，资产阶级赖以生产和占有产品的基础本身也就从它的脚下被挖掉了。它首先生产的是它自身的掘墓人。资产阶级的灭亡和无产阶级的胜利是同样不可避免的。
|
|
||||||
|
|
||||||
二、无产者和共产党人
|
|
||||||
|
|
||||||
共产党人同全体无产者的关系是怎样的呢？
|
|
||||||
|
|
||||||
共产党人不是同其他工人政党相对立的特殊政党。
|
|
||||||
|
|
||||||
他们没有任何同整个无产阶级的利益不同的利益。
|
|
||||||
|
|
||||||
他们不提出任何特殊的原则，用以塑造无产阶级的运动。
|
|
||||||
|
|
||||||
共产党人同其他无产阶级政党不同的地方只是：一方面，在无产者不同的民族的斗争中，共产党人强调和坚持整个无产阶级共同的不分民族的利益；另一方面，在无产阶级和资产阶级的斗争所经历的各个发展阶段上，共产党人始终代表整个运动的利益。
|
|
||||||
|
|
||||||
因此，在实践方面，共产党人是各国工人政党中最坚决的、始终起推动作用的部分；在理论方面，他们胜过其余无产阶级群众的地方在于他们了解无产阶级运动的条件、进程和一般结果。
|
|
||||||
|
|
||||||
共产党人的最近目的是和其他一切无产阶级政党的最近目的一样的：使无产阶级形成为阶级，推翻资产阶级的统治，由无产阶级夺取政权。
|
|
||||||
|
|
||||||
共产党人的理论原理，决不是以这个或那个世界改革家所发明或发现的思想、原则为根据的。
|
|
||||||
|
|
||||||
这些原理不过是现存的阶级斗争、我们眼前的历史运动的真实关系的一般表述。废除先前存在的所有制关系，并不是共产主义所独具的特征。
|
|
||||||
|
|
||||||
一切所有制关系都经历了经常的历史更替、经常的历史变更。
|
|
||||||
|
|
||||||
例如，法国革命废除了封建的所有制，代之以资产阶级的所有制。
|
|
||||||
|
|
||||||
共产主义的特征并不是要废除一般的所有制，而是要废除资产阶级的所有制。
|
|
||||||
|
|
||||||
但是，现代的资产阶级私有制是建立在阶级对立上面、建立在一些人对另一些人的剥削上面的产品生产和占有的最后而又完备的表现。
|
|
||||||
|
|
||||||
从这个意义上说，共产党人可以把自己的理论概括为一句话：消灭私有制。
|
|
||||||
|
|
||||||
有人责备我们共产党人，说我们消灭个人挣得的、自己劳动得来的财产，要消灭构成个人的一切自由、活动和独立的基础的财产。
|
|
||||||
|
|
||||||
好一个劳动得来的、自己挣得的、自己赚来的财产！你们说的是资产阶级财产出现以前的那种小资产阶级、小农的财产吗？那种财产用不着我们去消灭，工业的发展已经把它消灭了，而且每天都在消灭它。
|
|
||||||
|
|
||||||
或者，你们说的是现代的资产阶级的私有财产吧？
|
|
||||||
|
|
||||||
但是，难道雇佣劳动，无产者的劳动，会给无产者创造出财产来吗？没有的事。这种劳动所创造的资本，即剥削雇佣劳动的财产，只有在不断产生出新的雇佣劳动来重新加以剥削的条件下才能增殖的财产。现今的这种财产是在资本和雇佣劳动的对立中运动的。让我们来看看这种对立的两个方面吧。
|
|
||||||
|
|
||||||
做一个资本家，这就是说，他在生产中不仅占有一种纯粹个人的地位，而且占有一种社会地位。资本是集体的产物，它只有通过社会许多成员的共同活动，而且归根到底只有通过社会全体成员的共同活动，才能运动起来。
|
|
||||||
|
|
||||||
因此，资本不是一种个人力量，而是一种社会力量。
|
|
||||||
|
|
||||||
因此，把资本变为公共的、属于社会全体成员的财产，这并不是把个人财产变为社会财产。这里所改变的只是财产的社会性质。它将失掉它的阶级性质。
|
|
||||||
|
|
||||||
现在，我们来看看雇佣劳动。
|
|
||||||
|
|
||||||
雇佣劳动的平均价格是最低限度的工资，即工人为维持其工人的生活所必需的生活资料的数额。因此，雇佣工人靠自己的劳动所占有的东西，只够勉强维持他的生命的再生产。我们决不打算消灭这种供直接生命再生产用的劳动产品的个人占有，这种占有并不会留下任何剩余的东西使人们有可能支配别人的劳动。我们要消灭的只是这种占有的可怜的性质，在这种占有下，工人仅仅为增殖资本而活着，只有在统治阶级的利益需要他活着的时候才能活着。
|
|
||||||
|
|
||||||
在资产阶级社会里，活的劳动只是增殖已经积累起来的劳动的一种手段。在共产主义社会里，已经积累起来的劳动只是扩大、丰富和提高工人的生活的一种手段。
|
|
||||||
|
|
||||||
因此，在资产阶级社会里是过去支配现在，在共产主义社会里是现在支配过去。在资产阶级社会里，资本具有独立性和个性，而活动着的个人却没有独立性和个性。
|
|
||||||
|
|
||||||
而资产阶级却把消灭这种关系说成是消灭个性和自由！说对了。的确，正是要消灭资产者的个性、独立性和自由。
|
|
||||||
|
|
||||||
在现今的资产阶级生产关系的范围内，所谓自由就是自由贸易，自由买卖。
|
|
||||||
|
|
||||||
但是，买卖一消失，自由买卖也就会消失。关于自由买卖的言论，也像我们的资产阶级的其他一切关于自由的大话一样，仅仅对于不自由的买卖来说，对于中世纪被奴役的市民来说，才是有意义的，而对于共产主义要消灭买卖、消灭资产阶级生产关系和资产阶级本身这一点来说，却是毫无意义的。
|
|
||||||
|
|
||||||
我们要消灭私有制，你们就惊慌起来。但是，在你们的现存社会里，私有财产对十分之九的成员来说已经被消灭了；这种私有制这所以存在，正是因为私有财产对十分之九的成员来说已经不存在。可见，你们责备我们，是说我们要消灭那种以社会上的绝大多数人没有财产为必要条件的所有制。
|
|
||||||
|
|
||||||
总而言之，你们责备我们，是说我们要消灭你们的那种所有制。的确，我们是要这样做的。
|
|
||||||
|
|
||||||
从劳动不再能变为资本、货币、地租，一句话，不再能变为可以垄断的社会力量的时候起，就是说，从个人财产不再能变为资产阶级财产的时候起，你们说，个性被消灭了。
|
|
||||||
|
|
||||||
由此可见，你们是承认，你们所理解的个性，不外是资产者、资产阶级私有者。这样的个性确实应当被消灭。
|
|
||||||
|
|
||||||
共产主义并不剥夺任何人占有社会产品的权力，它只剥夺利用这种占有去奴役他人劳动的权力。
|
|
||||||
|
|
||||||
有人反驳说，私有制一消灭，一切活动就会停止，懒惰之风就会兴起。
|
|
||||||
|
|
||||||
这样说来，资产阶级社会早就应该因懒惰而灭亡了，因为在这个社会里劳者不获，获者不劳。所有这些顾虑，都可以归结为这样一个同义反复：一旦没有资本，也就不再有雇佣劳动了。
|
|
||||||
|
|
||||||
所有这些对共产主义的物质产品的占有方式和生产方式的责备，也被扩及到精神产品的占有和生产方面。正如阶级的所有制的终止在资产者看来是生产本身的终止一样，阶级的教育的终止在他们看来就等于一切教育的终止。
|
|
||||||
|
|
||||||
资产者唯恐失去的那种教育，绝大多数人来说是把人训练成机器。
|
|
||||||
|
|
||||||
但是，你们既然用你们资产阶级关于自由、教育、法等等的观念来衡量废除资产阶级所有制的主张，那就请你们不要同我们争论了。你们的观念本身是资产阶级的生产关系和所有制关系的产物，正像你们的法不过是被奉为法律的你们这个阶级的意志一样，而这种意志的内容是由你们这个阶级的物质生活条件决定的。
|
|
||||||
|
|
||||||
你们的利己观念使你们把自己的生产关系和所有制关系从历史的、在生产过程中是暂时的关系变成永恒的自然规律和理性规律，这种利己观念是你们和一切灭亡了的统治阶级所共有的。谈到古代所有制的时候你们所能理解的，谈到封建所有制的时候你们所能理解的，一谈到资产阶级所有制你们就再也不能理解了。
|
|
||||||
|
|
||||||
消灭家庭！连极端的激进派也对共产党人的这种可耻的意图表示愤慨。
|
|
||||||
|
|
||||||
现代的、资产阶级的家庭是建立在什么基础上的呢？是建立在资本上面，建立在私人发财上面的。这种家庭只是在资产阶级那里才以充分发展的形式存在着，而无产者的被迫独居和公开的卖淫则是它的补充。
|
|
||||||
|
|
||||||
资产者的家庭自然会随着它的这种补充的消失而消失，两者都要随着资本的消失而消失。
|
|
||||||
|
|
||||||
你们是责备我们要消灭父母对子女的剥削吗？我们承认这种罪状。
|
|
||||||
|
|
||||||
但是，你们说，我们用社会教育代替家庭教育，就是要消灭人们最亲密的关系。
|
|
||||||
|
|
||||||
而你们的教育不也是由社会决定的吗？不也是由你们进行教育时所处的那种社会关系决定的吗？不也是由社会通过学校等等进行的直接的或间接的干涉决定的吗？共产党人并没有发明社会对教育的作用；他们仅仅是要改变这种作用的性质，要使教育摆脱统治阶级的影响。
|
|
||||||
|
|
||||||
无产者的一切家庭联系越是由于大工业的发展而被破坏，他们的子女越是由于这种发展而被变成单纯的商品和劳动工具，资产阶级关于家庭和教育、关于父母和子女的亲密关系的空话就越是令人作呕。
|
|
||||||
|
|
||||||
但是，你们共产党人是要实行公妻制的啊，——整个资产阶级异口同声地向我们这样叫喊。
|
|
||||||
|
|
||||||
资产者是把自己的妻子看作单纯的生产工具的。他们听说生产工具将要公共使用，自然就不能不想到妇女也会遭到同样的命运。
|
|
||||||
|
|
||||||
他们想也没有想到，问题正在于使妇女不再处于单纯生产工具的地位。
|
|
||||||
|
|
||||||
其实，我们的资产者装得道貌岸然，对所谓的共产党人的正式公妻制表示惊讶，那是再可笑不过了。公妻制无需共产党人来实行，它差不多是一向就有的。
|
|
||||||
|
|
||||||
我们的资产者不以他们的无产者的妻子和女儿受他们支配为满足，正式的卖淫更不必说了，他们还以互相诱奸妻子为最大的享乐。
|
|
||||||
|
|
||||||
资产阶级的婚姻实际上是公妻制。人们至多只能责备共产党人，说他们想用正式的、公开的公妻制来代替伪善地掩蔽着的公妻制。其实，不言而喻，随着现在的生产关系的消灭，从这种关系中产生的公妻制，即正式的和非正式的卖淫，也就消失了。
|
|
||||||
|
|
||||||
有人还责备共产党人，说他们要取消祖国，取消民族。
|
|
||||||
|
|
||||||
工人没有祖国。决不能剥夺他们所没有的东西。因为无产阶级首先必须取得政治统治，上升为民族的阶级，把自身组织成为民族，所以它本身还是民族的，虽然完全不是资产阶级所理解的那种意思。
|
|
||||||
|
|
||||||
随着资产阶级的发展，随着贸易自由的实现和世界市场的建立，随着工业生产以及与之相适应的生活条件的趋于一致，各国人民之间的民族分隔和对立日益消失。
|
|
||||||
|
|
||||||
无产阶级的统治将使它们更快地消失。联合的行动，至少是各文明国家的联合的行动，是无产阶级获得解放的首要条件之一。
|
|
||||||
|
|
||||||
人对人的剥削一消灭，民族对民族的剥削就会随之消灭。
|
|
||||||
|
|
||||||
民族内部的阶级对立一消失，民族之间的敌对关系就会随之消失。
|
|
||||||
|
|
||||||
从宗教的、哲学的和一切意识形态的观点对共产主义提出的种种责难，都不值得详细讨论了。
|
|
||||||
|
|
||||||
人们的观念、观点和概念，一句话，人们的意识，随着人们的生活条件、人们的社会关系、人们的社会存在的改变而改变，这难道需要经过深思才能了解吗？
|
|
||||||
|
|
||||||
思想的历史除了证明精神生产随着物质生产的改造而改造，还证明了什么呢？任何一个时代的统治思想始终都不过是统治阶级的思想。
|
|
||||||
|
|
||||||
当人们谈到使整个社会革命化的思想时，他们只是表明了一个事实：在旧社会内部已经形成了新社会的因素，旧思想的瓦解是同旧生活条件的瓦解步调一致的。
|
|
||||||
|
|
||||||
当古代世界走向灭亡的时候，古代的各种宗教就被基督教战胜了。当基督教思想在18世纪被启蒙思想击败的时候，封建社会正在同当时革命的资产阶级进行殊死的斗争。信仰自由和宗教自由的思想，不过表明竞争在信仰领域里占统治地位罢了。
|
|
||||||
|
|
||||||
“但是”，有人会说，“宗教的、道德的、哲学的、政治的、法的观念等等在历史发展的进程中固然是不断改变的，而宗教、道德、哲学、政治和法在这种变化中却始终保存着。
|
|
||||||
|
|
||||||
此外，还存在着一切社会状态所共有的永恒真理，如自由、正义等等。但是共产主义要废除永恒真理，它要废除宗教、道德，而不是加以革新，所以共产主义是同至今的全部历史发展相矛盾的。”
|
|
||||||
|
|
||||||
这种责难归结为什么呢？至今的一切社会的历史都是在阶级对立中运动的，而这种对立在不同的时代具有不同的形式。
|
|
||||||
|
|
||||||
但是，不管阶级对立具有什么样的形式，社会上一部分人对另一部分人的剥削却是过去各个世纪所共有的事实。因此，毫不奇怪，各个世纪的社会意识，尽管形形色色、千差万别，总是在某些共同的形式中运动的，这些形式，这些意识形式，只有当阶级对立完全消失的时候才会完全消失。
|
|
||||||
|
|
||||||
共产主义革命就是同传统的所有制关系实行最彻底的决裂；毫不奇怪，它在自己的发展进程中要同传统的观念实行最彻底的决裂。
|
|
||||||
|
|
||||||
不过，我们还是把资产阶级对共产主义的种种责难撇开吧。
|
|
||||||
|
|
||||||
前面我们已经看到，工人革命的第一步就是使无产阶级上升为统治阶级，争得民主。
|
|
||||||
|
|
||||||
无产阶级将利用自己的政治统治，一步一步地夺取资产阶级的全部资本，把一切生产工具集中在国家即组织成为统治阶级的无产阶级手里，并且尽可能快地增加生产力的总量。
|
|
||||||
|
|
||||||
要做到这一点，当然首先必须对所有权和资产阶级生产关系实行强制性的干涉，也就是采取这样一些措施，这些措施在经济上似乎是不够充分的和没有力量的，但是在运动进程中它们会越出本身，而且作为变革全部生产方式的手段是必不可少的。
|
|
||||||
|
|
||||||
这些措施在不同的国家里当然会是不同的。
|
|
||||||
|
|
||||||
但是，最先进的国家几乎都可以采取下面的措施：
|
|
||||||
|
|
||||||
1、剥夺地产，把地租用于国家支出。
|
|
||||||
|
|
||||||
2、征收高额累进税。
|
|
||||||
|
|
||||||
3、废除继承权。
|
|
||||||
|
|
||||||
4、没收一切流亡分子和叛乱分子的财产。
|
|
||||||
|
|
||||||
5、通过拥有国家资本和独享垄断权的国家银行，把信贷集中在国家手里。
|
|
||||||
|
|
||||||
6、把全部运输业集中在国家的手里。
|
|
||||||
|
|
||||||
7、按照总的计划增加国家工厂和生产工具，开垦荒地和改良土壤。
|
|
||||||
|
|
||||||
8、实行普遍劳动义务制，成立产业军，特别是在农业方面。
|
|
||||||
|
|
||||||
9、把农业和工业结合起来，促使城乡对立逐步消灭。
|
|
||||||
|
|
||||||
10、对所有儿童实行公共的和免费的教育。取消现在这种形式的儿童的工厂劳动。把教育同物质生产结合起来，等等。
|
|
||||||
|
|
||||||
当阶级差别在发展进程中已经消失而全部生产集中在联合起来的个人的手里的时候，公共权力就失去政治性质。原来意义上的政治权力，是一个阶级用以压迫另一个阶级的有组织的暴力。如果说无产阶级在反对资产阶级的斗争中一定要联合为阶级，如果说它通过革命使自己成为统治阶级，并以统治阶级的资格用暴力消灭旧的生产关系，那么它在消灭这种生产关系的同时，也就消灭了阶级对立的存在条件，消灭阶级本身的存在条件，从而消灭了它自己这个阶级的统治。
|
|
||||||
|
|
||||||
代替那存在着阶级和阶级对立的资产阶级旧社会的，将是这样一个联合体，在那里，每个人的自由发展是一切人的自由发展的条件。
|
|
||||||
|
|
||||||
三、社会主义的和共产主义的文献
|
|
||||||
|
|
||||||
1．反动的社会主义
|
|
||||||
|
|
||||||
（甲）封建的社会主义
|
|
||||||
|
|
||||||
法国和英国的贵族，按照他们的历史地位所负的使命，就是写一些抨击现代资产阶级社会的作品。在法国的1830年七月革命和英国的改革运动 中，他们再一次被可恨的暴发户打败了。从此就再谈不上严重的政治斗争了。他们还能进行的只是文字斗争。但是，即使在文字方面也不可能重弹复辟时期的老调了。为了激起同情，贵族们不得不装模作样，似乎他们已经不关心自身的利益，只是为了被剥削的工人阶级的利益才去写对资产阶级的控诉书。他们用来泄愤的手段是：唱唱诅咒他们的新统治者的歌，并向他叽叽咕咕地说一些或多或少凶险的预言。
|
|
||||||
|
|
||||||
这样就产生了封建的社会主义，半是挽歌，半是谤文，半是过去的回音，半是未来的恫吓；它有时也能用辛辣、俏皮而尖刻的评论剌中资产阶级的心，但是它由于完全不能理解现代历史的进程而总是令人感到可笑。
|
|
||||||
|
|
||||||
为了拉拢人民，贵族们把无产阶级的乞食袋当作旗帜来挥舞。但是，每当人民跟着他们走的时候，都发现他们的臀部带有旧的封建纹章，于是就哈哈大笑，一哄而散。
|
|
||||||
|
|
||||||
一部分法国正统派和“青年英国”，都演过这出戏。
|
|
||||||
|
|
||||||
封建主说，他们的剥削方式和资产阶级的剥削不同，那他们只是忘记了，他们是在完全不同的、目前已经过时的情况和条件下进行剥削的。他们说，在他们的统治下并没有出现过现代的无产阶级，那他们只是忘记了，现代的资产阶级正是他们的社会制度的必然产物。
|
|
||||||
|
|
||||||
不过，他们毫不掩饰自己的批评的反动性质，他们控告资产阶级的主要罪状正是在于：在资产阶级的统治下有一个将把整个旧社会制度炸毁的阶级发展起来。
|
|
||||||
|
|
||||||
他们责备资产阶级，与其说是因为它产生了无产阶级，不如说是因为它产生了革命的无产阶级。
|
|
||||||
|
|
||||||
因此，在政治实践中，他们参与对工人阶级采取的一切暴力措施，在日常生活中，他们违背自己的那一套冠冕堂皇的言词，屈尊拾取金苹果，不顾信义、仁爱和名誉去做羊毛、甜菜和烧洒的买卖。
|
|
||||||
|
|
||||||
正如僧侣总是同封建主携手同行一样，僧侣的社会主义也总是同封建的社会主义携手同行的。
|
|
||||||
|
|
||||||
要给基督教禁欲主义涂上一层社会主义的色彩，是再容易不过了。基督教不是也激烈反对私有财产，反对婚姻，反对国家吗？它不是提倡用行善和求乞、独身和禁欲、修道和礼拜来代替这一切吗？基督教的社会主义，只不过是僧侣用来使贵族的怨愤神圣的圣水罢了。
|
|
||||||
|
|
||||||
（乙）小资产阶级的社会主义
|
|
||||||
|
|
||||||
封建贵族并不是被资产阶级所推翻的、其生活条件在现代资产阶级社会里日益恶化和消失的唯一阶级。中世纪的城关市民和小农等级是现代资产阶级的前身。在工商业不很发达的国家里，这个阶级还在新兴的资产阶级身旁勉强生存着。
|
|
||||||
|
|
||||||
在现代文明已经发展的国家里，形成了一个新的小资产阶级，它摇摆于无产阶级和资产阶级之间，并且作为资产阶级社会的补充部分不断地重新组成。但是，这一阶级的成员经常被竞争抛到无产阶级队伍里去，而且，随着大工业的发展，他们甚至觉察到，他们很快就会完全失去他们作为现代社会中一个独立部分的地位，在商业、工业和农业中很快就会被监工和雇员所代替。
|
|
||||||
|
|
||||||
在农民阶级远远超过人口半数的国家，例如在法国，那些站在无产阶级方面反对资产阶级的著作家，自然是用小资产阶级和小农的尺度去批判资产阶级制度的，是从小资产阶级的立场出发替工人说话的。这样就形成了小资产阶级的社会主义。西斯蒙第不仅对法国而且对英国来说都是这类著作家的首领。
|
|
||||||
|
|
||||||
这种社会主义非常透彻地分析了现代生产关系中的矛盾。它揭穿了经济学家的虚伪的粉饰。它确凿地证明了机器和分工的破坏作用、资本和地产的积聚、生产过剩、危机、小资产者和小农的必然没落、无产阶级的贫困、生产的无政府状态、财富分配的极不平均、各民族之间的毁灭性的工业战争，以及旧风尚、旧家庭关系和旧民族性的解体。
|
|
||||||
|
|
||||||
但是，这种社会主义按其实际内容来说，或者是企图恢复旧的生产资料和交换手段，从而恢复旧的所有制关系和旧的社会，或者是企图重新把现代的生产资料和交换手段硬塞到已被它们突破而且必然被突破的旧的所有制关系的框子里去。它在这两种场合都是反动的，同时又是空想的。
|
|
||||||
|
|
||||||
工业中的行会制度，农业中的宗法经济，——这就是它的结论。
|
|
||||||
|
|
||||||
这一思潮在它以后的发展中变成了一种怯懦的悲叹。
|
|
||||||
|
|
||||||
（丙）德国的或“真正的”社会主义
|
|
||||||
|
|
||||||
法国的社会主义和共产主义的文献是在居于统治地位的资产阶级的压迫下产生的，并且是同这种统治作斗争的文字表现，这种文献被搬到德国的时候，那里的资产阶级才刚刚开始进行反对封建专制制度的斗争。
|
|
||||||
|
|
||||||
德国的哲学家、半哲学家和美文学家，贪婪地抓住了这种文献，不过他们忘记了：在这种著作从法国搬到德国的时候，法国的生活条件却没有同时搬过去。在德国的条件下，法国的文献完全失去了直接实践的意义，而只具有纯粹文献的形式。它必然表现为关于真正的社会、关于实现人的本质的无谓思辨。这样，第一次法国革命的要求，在18世纪的德国哲学家看来，不过是一般“实践理性”的要求，而革命的法国资产阶级的意志的表现，在他们心目中就是纯粹的意志、本来的意志、真正人的意志的规律。
|
|
||||||
|
|
||||||
德国著作家的唯一工作，就是把新的法国的思想同他们的旧的哲学信仰调和起来，或者毋宁说，就是从他们的哲学观点出发去掌握法国的思想。
|
|
||||||
|
|
||||||
这种掌握，就像掌握外国语一样，是通过翻译的。
|
|
||||||
|
|
||||||
大家知道，僧侣们曾经在古代异教经典的手抄本上面写上荒诞的天主教圣徒传。德国著作家对世俗的法国文献采取相反的作法。他们在法国的原著下面写上自己的哲学胡说。例如，他们在法国人对货币关系的批判下面写上“人的本质的外化”，在法国人对资产阶级国家的批判下面写上所谓“抽象普遍物的统治的扬弃”，等等。
|
|
||||||
|
|
||||||
这种在法国人的论述下面塞进自己哲学词句的做法，他们称之为“行动的哲学”、”真正的社会主义”、“德国的社会主义科学”、“社会主义的哲学论证”，等等。
|
|
||||||
|
|
||||||
法国的社会主义和共产主义的文献就这样被完全阉割了。既然这种文献在德国人手里已不再表现一个阶级反对另一个阶级的斗争，于是德国人就认为：他们克服了“法国人的片面性”，他们不代表真实的要求，而代表真理的要求，不代表无产者的利益，而代表人的本质的利益，即一般人的利益，这种人不属于任何阶级，根本不存在于现实界，而只存在于云雾弥漫的哲学幻想的太空。
|
|
||||||
|
|
||||||
这种曾经郑重其事地看待自己那一套拙劣的小学生作业并且大言不惭地加以吹嘘的德国社会主义，现在渐渐失去了它的自炫博学的天真。
|
|
||||||
|
|
||||||
德国的特别是普鲁士的资产阶级反对封建主和专制王朝的斗争，一句话，自由主义运动，越来越严重了。
|
|
||||||
|
|
||||||
于是，“真正的”社会主义就得到了一个好机会，把社会主义的要求同政治运动对立起来，用诅咒异端邪说的传统办法诅咒自由主义，诅咒代议制国家，诅咒资产阶级的竞争、资产阶级的新闻出版自由、资产阶级的法、资产阶级的自由和平等，并且向人民群众大肆宣扬，说什么在这个资产阶级运动中，人民群众非但一无所得，反而会失去一切。德国的社会主义恰好忘记了，法国的批判（德国的社会主义是这种批判的可怜的回声）是以现代的资产阶级社会以及相应的物质生活条件和相当的政治制度为前提的，而这一切前提当时在德国正是尚待争取的。
|
|
||||||
|
|
||||||
这种社会主义成了德意志各邦专制政府及其随从——僧侣、教员、容克和官僚求之不得的、吓唬来势汹汹的资产阶级的稻草人。
|
|
||||||
|
|
||||||
这种社会主义是这些政府用来镇压德国工人起义的毒辣的皮鞭和枪弹的甜蜜的补充。
|
|
||||||
|
|
||||||
既然“真正的”社会主义就这样成了这些政府对付德国资产阶级的武器，那么它也就直接代表了一种反动的利益，即德国小市民的利益。在德国，16世纪遗留下来的、从那时起经常以不同形式重新出现的小资产阶级，是现存制度的真实的社会基础。
|
|
||||||
|
|
||||||
保存这个小资产阶级，就是保存德国的现存制度。这个阶级胆战心惊地从资产阶级的工业统治和政治统治那里等候着无可幸免的灭亡，这一方面是由于资本的积聚，另一方面是由于革命无产阶级的兴起。在它看来，“真正的”社会主义能起一箭双雕的作用。“真正的”社会主义像瘟疫一样流行起来了。
|
|
||||||
|
|
||||||
德国的社会主义者给自己的那几条干瘪的“永恒真理”披上一件用思辨的蛛丝织成的、绣满华丽辞藻的花朵和浸透甜情蜜意的甘露的外衣，这件光彩夺目的外衣只是使他们的货物在这些顾客中间增加销路罢了。
|
|
||||||
|
|
||||||
同时，德国的社会主义也越来越认识到自己的使命就是充当这种小市民的夸夸其谈的代言人。
|
|
||||||
|
|
||||||
它宣布德意志民族是模范的民族，德国小市民是模范的人。它给这些小市民的每一种丑行都加上奥秘的、高尚的、社会主义的意义，使之变成完全相反的东西。它发展到最后，就直接反对共产主义的“野蛮破坏的”倾向，并且宣布自己是不偏不倚地超乎任何阶级斗争之上的。现今在德国流行的一切所谓社会主义和共产主义的著作，除了极少数的例外，都属于这一类卑鄙龌龊的、令人委靡的文献。
|
|
||||||
|
|
||||||
2．保守的或资产阶级的社会主义
|
|
||||||
|
|
||||||
资产阶级中的一部分人想要消除社会的弊病，以便保障资产阶级社会的生存。
|
|
||||||
|
|
||||||
这一部分人包括：经济学家、博爱主义者、人道主义者、劳动阶级状况改善派、慈善事业组织者、动物保护协会会员、戒酒协会发起人以及形形色色的小改良家。这种资产阶级的社会主义甚至被制成一些完整的体系。
|
|
||||||
|
|
||||||
我们可以举蒲鲁东的《贫困的哲学》作为例子。
|
|
||||||
|
|
||||||
社会主义的资产者愿意要现代社会的生存条件，但是不要由这些条件必然产生的斗争和危险。他们愿意要现存的社会，但是不要那些使这个社会革命化和瓦解的因素。他们愿意要资产阶级，但是不要无产阶级。在资产阶级看来，它所统治的世界自然是最美好的世界。资产阶级的社会主义把这种安慰人心的观念制成半套或整套的体系。它要求无产阶级实现它的体系，走进新的耶路撒冷，其实它不过是要求无产阶级停留在现今的社会里，但是要抛弃他们关于这个社会的可恶的观念。
|
|
||||||
|
|
||||||
这种社会主义的另一种不够系统、但是比较实际的形式，力图使工人阶级厌弃一切革命运动，硬说能给工人阶级带来好处的并不是这样或那样的政治改革，而仅仅是物质生活条件即经济关系的改变。但是，这种社会主义所理解的物质生活条件的改变，绝对不是只有通过革命的途径才能实现的资产阶级生产关系的废除，而是一些在这种生产关系的基础上实行的行政上的改良，因而丝毫不会改变资本和雇佣劳动的关系，至多只能减少资产阶级的统治费用和简化它的财政管理。
|
|
||||||
|
|
||||||
资产阶级的社会主义只有在它变成纯粹的演说辞令的时候，才获得自己的适当的表现。
|
|
||||||
|
|
||||||
自由贸易！为了工人阶级的利益；保护关税！为了工人阶级的利益；单身牢房！为了工人阶级的利益。——这才是资产阶级的社会主义唯一认真说出的最后的话。
|
|
||||||
|
|
||||||
资产阶级的社会主义就是这样一个论断：资产者之为资产者，是为了工人阶级的利益。
|
|
||||||
|
|
||||||
3．批判的空想的社会主义和共产主义
|
|
||||||
|
|
||||||
在这里，我们不谈在现代一切大革命中表达过无产阶级要求的文献（巴贝夫等人的著作）。
|
|
||||||
|
|
||||||
无产阶级在普遍激动的时代、在推翻封建社会的时期直接实现自己阶级利益的最初尝试，都不可避免地遭到了失败，这是由于当时无产阶级本身还不够发展，由于无产阶级解放的物质条件还没具备，这些条件只是资产阶级时代的产物。随着这些早期的无产阶级运动而出现的革命文献，就其内容来说必然是反动的。这种文献倡导普遍的禁欲主义和粗陋的平均主义。
|
|
||||||
|
|
||||||
本来意义的社会主义和共产主义的体系，圣西门、傅立叶、欧文等人的体系，是在无产阶级和资产阶级之间的斗争还不发展的最初时期出现的。关于这个时期，我们在前面已经叙述过了（见《资产阶级和无产阶级》）。
|
|
||||||
|
|
||||||
诚然，这些体系的发明家看到了阶级的对立，以及占统治地位的社会本身中的瓦解因素的作用。但是，他们看不到无产阶级方面的任何历史主动性，看不到它所特有的任何政治运动。
|
|
||||||
|
|
||||||
由于阶级对立的发展是同工业的发展步调一致的，所以这些发明家也不可能看到无产阶级解放的物质条件，于是他们就去探求某种社会科学、社会规律，以便创造这些条件。
|
|
||||||
|
|
||||||
社会的活动要由他们个人的发明活动来代替，解放的历史条件要由幻想的条件来代替，无产阶级的逐步组织成为阶级要由一种特意设计出来的社会组织来代替。在他们看来，今后的世界历史不过是宣传和实施他们的社会计划。
|
|
||||||
|
|
||||||
诚然，他们也意识到，他们的计划主要是代表工人阶级这一受苦最深的阶级的利益。在他们心目中，无产阶级只是一个受苦最深的阶级。
|
|
||||||
|
|
||||||
但是，由于阶级斗争不发展，由于他们本身的生活状况，他们就以为自己是高高超乎这种阶级对立之上的。他们要改善社会一切成员的生活状况，甚至生活最优裕的成员也包括在内。因此，他们总是不加区别地向整个社会呼吁，而且主要是向统治阶级呼吁。他们以为，人们只要理解他们的体系，就会承认这种体系是最美好的社会的最美好的计划。
|
|
||||||
|
|
||||||
因此，他们拒绝一切政治行动，特别是一切革命行动；他们想通过和平的途径达到自己的目的，并且企图通过一些小型的、当然不会成功的试验，通过示范的力量来为新的社会福音开辟道路。
|
|
||||||
|
|
||||||
这种对未来社会的幻想的描绘，在无产阶级还很不发展、因而对本身的地位的认识还基于幻想的时候，是同无产阶级对社会普遍改造的最初的本能的渴望相适应的。
|
|
||||||
|
|
||||||
但是，这些社会主义和共产主义的著作也含有批判的成分。这些著作抨击现存社会的全部基础。因此，它们提供了启发工人觉悟的极为宝贵的材料。它们关于未来社会的积极的主张，例如消灭城乡对立，消灭家庭，消灭私人营利，消灭雇佣劳动，提倡社会和谐，把国家变成纯粹的生产管理机构，——所有这些主张都只是表明要消灭阶级对立，而这种阶级对立在当时刚刚开始发展，它们所知道的只是这种对立的早期的、不明显的、不确定的形式。因此，这些主张本身还带有纯粹空想的性质。
|
|
||||||
|
|
||||||
批判的空想的社会主义和共产主义的意义，是同历史的发展成反比的。阶级斗争越发展和越具有确定的形式，这种超乎阶级斗争的幻想，这种反对阶级斗争的幻想，就越失去任何实践意义和任何理论根据。所以，虽然这些体系的创始人在许多方面是革命的，但是他们的信徒总是组成一些反动的宗派。这些信徒无视无产阶级的历史进展，还是死守着老师们的旧观点。因此，他们一贯企图削弱阶级斗争，调和对立。他们还总是梦想用试验的办法来实现自己的社会空想，创办单个的法伦斯泰尔，建立国内移民区，创立小伊加利亚，即袖珍版的新耶路撒冷，——而为了建造这一切空中楼阁，他们就不得不呼吁资产阶级发善心和慷慨解囊。他们逐渐地堕落到上述反动的或保守的社会主义者的一伙中去了，所不同的只是他们更加系统地卖弄学问，狂热地迷信自己那一套社会科学的奇功异效。
|
|
||||||
|
|
||||||
因此，他们激烈地反对工人的一切政治运动，认为这种运动只是由于盲目地不相信新福音才发生的。
|
|
||||||
|
|
||||||
在英国，有欧文主义者反对宪章派，在法国，有傅立叶主义者反对改革派。
|
|
||||||
|
|
||||||
四、共产党人对各种反对党派的态度
|
|
||||||
|
|
||||||
看过第二章之后，就可以了解共产党人同已经形成的工人政党的关系，因而也就可以了解他们同英国宪章派和北美土地改革派的关系。
|
|
||||||
|
|
||||||
共产党人为工人阶级的最近的目的和利益而斗争，但是他们在当前的运动中同时代表运动的未来。在法国，共产党人同社会主义民主党联合起来反对保守的和激进的资产阶级，但是并不因此放弃对那些从革命的传统中承袭下来的空谈和幻想采取批判态度的权利。
|
|
||||||
|
|
||||||
在瑞士，共产党人支持激进派，但是并不忽略这个政党是由互相矛盾的分子组成的，其中一部分是法国式的民主社会主义者，一部分是激进的资产者。
|
|
||||||
|
|
||||||
在波兰人中间，共产党人支持那个把土地革命当作民族解放的条件的政党，即发动过1846年克拉科夫起义的政党。
|
|
||||||
|
|
||||||
在德国，只要资产阶级采取革命的行动，共产党就同它一起去反对专制君主制、封建土地所有制和小市民的反动性。
|
|
||||||
|
|
||||||
但是，共产党一分钟也不忽略教育工人尽可能明确地意识到资产阶级和无产阶级的敌对的对立，以便德国工人能够立刻利用资产阶级统治所必然带来的社会的和政治的条件作为反对资产阶级的武器，以便在推翻德国的反动阶级之后立即开始反对资产阶级本身的斗争。
|
|
||||||
|
|
||||||
共产党人把自己的主要注意力集中在德国，因为德国正处在资产阶级革命的前夜，因为同17世纪的英国和18世纪的法国相比，德国将在整个欧洲文明更进步的条件下，拥有发展得多的无产阶级去实现这个变革，因而德国的资产阶级革命只能是无产阶级革命的直接序幕。
|
|
||||||
|
|
||||||
总之，共产党人到处都支持一切反对现存的社会制度和政治制度的革命运动。
|
|
||||||
|
|
||||||
在所有这些运动中，他们都强调所有制问题是运动的基本问题，不管这个问题的发展程度怎样。
|
|
||||||
|
|
||||||
最后，共产党人到处都努力争取全世界民主政党之间的团结和协调。
|
|
||||||
|
|
||||||
共产党人不屑于隐瞒自己的观点和意图。他们公开宣布：他们的目的只有用暴力推翻全部现存的社会制度才能达到。让统治阶级在共产主义革命面前发抖吧。无产者在这个革命中失去的只是锁链。他们获得的将是整个世界。
|
|
||||||
|
|
||||||
全世界无产者，联合起来！
|
|
||||||
""")
|
|
||||||
|
|
||||||
query = '福娘的物种'
|
|
||||||
nodes = rag_worker.retrieve_from_store_with_query(query)
|
|
||||||
build_prompt = rag_worker.build_prompt(query, nodes)
|
|
||||||
preview = rag_worker.generate_node_array_preview(nodes)
|
|
||||||
print(preview)
|
|
||||||
print(build_prompt)
|
|
||||||
print(nodes)
|
|
||||||
|
|
||||||
# vs = rag_worker.load_from_checkpoint('./good_man_vector_store')
|
|
||||||
# rag_worker.add_text_to_vector_store(r"I see that the (0.6.0) index persisted on disk contains: docstore.json, index_store.json and vector_store.json, but they don't seem to contain file paths or title metadata from the original documents, so maybe that's not captured and stored?")
|
|
||||||
# rag_worker.add_text_to_vector_store(r"Thanks! I'm trying to cluster (all) the vectors, then generate a description (label) for each cluster by sending (just) the vectors in each cluster to GPT to summarize, then associate the vectors with the original documents and classify each document by applying a sort of weighted sum of its cluster-labeled snippets. Not sure how useful that will be, but I want to try! I've got the vectors now (although I'm bit worried that the nested structure I'm getting them from might change without warning in the future!), and I'm able to cluster them, but I don't know how to associate the vectors (via their nodes) back to the original documents yet...")
|
|
||||||
# res = rag_worker.retrieve_from_store_with_query('cluster')
|
|
||||||
# rag_worker.save_to_checkpoint(checkpoint_dir = './good_man_vector_store')
|
|
||||||
|
|
||||||
# print(vs)
|
|
||||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user