生而为人

Untitled

Posted on 2026-05-05

Microsoft Windows [Version 10.0.19044.2006]
(c) Microsoft Corporation. All rights reserved.

C:\Users\jingqicao>echo PATH
PATH

C:\Users\jingqicao>echo $PATH
$PATH

C:\Users\jingqicao>echo %PATH
%PATH

C:\Users\jingqicao>echo %PATH%
C:\Program Files (x86)\Common Files\Oracle\Java\javapath;C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\wbin;C:\Program Files\Common Files\Microsoft Shared\Microsoft Online Services;C:\Program Files (x86)\Common Files\Microsoft Shared\Microsoft Online Services;C:\windows\system32;C:\windows;C:\windows\System32\Wbem;C:\windows\System32\WindowsPowerShell\v1.0\;C:\windows\System32\OpenSSH\;C:\Program Files\Microsoft SQL Server\130\Tools\Binn\;C:\Program Files\Microsoft SQL Server\Client SDK\ODBC\170\Tools\Binn\;C:\Program Files\Git\cmd;C:\Program Files\Go\bin;D:\Support\gawk-3.1.6-1-bin\bin;D:\Support\Nuget;C:\Program Files\dotnet\;C:\Users\jingqicao\AppData\Local\CloudBuild;C:\Users\jingqicao\AppData\Local\CloudBuild;C:\Users\jingqicao\AppData\Local\Microsoft\WindowsApps;C:\Users\jingqicao\.dotnet\tools;C:\Users\jingqicao\AppData\Local\Programs\Microsoft VS Code\bin;C:\Users\jingqicao\Program Files\protoc-3.13.0-win64\bin;D:\Support\spark-3.0.1-bin-hadoop2.7\bin;D:\Support\spark-3.0.1-bin-hadoop2.7\sbin;C:\Program Files\Java\jdk1.8.0_291\bin;D:\Support\hadoop-2.7.7\bin;D:\Support\hadoop-2.7.7\sbin;D:\Support\apache-maven-3.8.1\bin;C:\Users\jingqicao\AppData\Local\Programs\Python\Python39\Scripts;C:\Users\jingqicao\AppData\Local\atom\bin;C:\Users\jingqicao\.dotnet\tools;D:\Support\Go\bin;

C:\Users\jingqicao>protoc.exe -I=D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\ --go_out=D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas.Go\Events\ --go_opt=file_extension=.pd.go D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\Events\App.proto
protoc-gen-go: no such flag -file_extension
--go_out: protoc-gen-go: Plugin failed with status code 1.

C:\Users\jingqicao>protoc.exe -I=D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\ --go_out=D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas.Go\Events\ --go_opt=.pd.go D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\Events\App.proto
protoc-gen-go: no such flag -.pd.go
--go_out: protoc-gen-go: Plugin failed with status code 1.

C:\Users\jingqicao>protoc.exe -I=D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\ --go_out=D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas.Go\Events\ --go_opt=. D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\Events\App.proto
protoc-gen-go: no such flag -.
--go_out: protoc-gen-go: Plugin failed with status code 1.

C:\Users\jingqicao>protoc.exe -I=D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\ --go_out=D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas.Go\Events\ D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\Events\App.proto
protoc-gen-go: unable to determine Go import path for "Events/App.proto"

Please specify either:
        • a "go_package" option in the .proto source file, or
        • a "M" argument on the command line.

See https://developers.google.com/protocol-buffers/docs/reference/go-generated#package for more information.

--go_out: protoc-gen-go: Plugin failed with status code 1.

C:\Users\jingqicao>protoc.exe -I=D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\ --go_out=D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas.Go\Events\ D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\Events\App.proto D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\Events\Auction.proto D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\Events\AuctionFacts.proto D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\Events\Bid.proto D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\Events\ClickEvent.proto D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\Events\CommonTypes.proto D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\Events\Device.proto D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\Events\ExternalUser.proto D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\Events\L1ClassifierScore.proto D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\Events\Location.proto D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\Events\MediationConfigParams.proto D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\Events\MediationServingLogRecord.proto D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\Events\OperatingSystem.proto D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\Events\PiiRecord.proto D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\Events\Revenue.proto D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\Events\Site.proto D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\Events\User.proto D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\Events\ViewabilityEvent.proto
google/protobuf/timestamp.proto: File not found.
Events/ViewabilityEvent.proto:4:1: Import "google/protobuf/timestamp.proto" was not found or had errors.
Events/ViewabilityEvent.proto:21:5: "google.protobuf.Timestamp" is not defined.
Events/ClickEvent.proto:4:1: Import "google/protobuf/timestamp.proto" was not found or had errors.
Events/ClickEvent.proto:18:9: "google.protobuf.Timestamp" is not defined.
Events/AuctionFacts.proto:3:1: Import "google/protobuf/timestamp.proto" was not found or had errors.
Events/AuctionFacts.proto:10:1: Import "Events/ViewabilityEvent.proto" was not found or had errors.
Events/AuctionFacts.proto:11:1: Import "Events/ClickEvent.proto" was not found or had errors.
Events/AuctionFacts.proto:23:9: "google.protobuf.Timestamp" is not defined.
Events/AuctionFacts.proto:45:9: "ViewabilityEvent" is not defined.
Events/AuctionFacts.proto:46:9: "ViewabilityEvent" is not defined.
Events/AuctionFacts.proto:47:9: "ClickEvent" is not defined.

C:\Users\jingqicao>protoc.exe -I=D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\ -I=D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\ --go_out=D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas.Go\Events\ D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\Events\App.proto
protoc-gen-go: unable to determine Go import path for "Events/App.proto"

Please specify either:
        • a "go_package" option in the .proto source file, or
        • a "M" argument on the command line.

See https://developers.google.com/protocol-buffers/docs/reference/go-generated#package for more information.

--go_out: protoc-gen-go: Plugin failed with status code 1.

C:\Users\jingqicao>protoc.exe -I=C:\Users\jingqicao\.nuget\packages\google.protobuf.tools\3.6.1\tools -I=D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\ --go_out=D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas.Go\Events\ D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\Events\App.proto
protoc-gen-go: unable to determine Go import path for "Events/App.proto"

Please specify either:
        • a "go_package" option in the .proto source file, or
        • a "M" argument on the command line.

See https://developers.google.com/protocol-buffers/docs/reference/go-generated#package for more information.

--go_out: protoc-gen-go: Plugin failed with status code 1.

C:\Users\jingqicao>protoc.exe -I=C:\Users\jingqicao\.nuget\packages\google.protobuf.tools\3.6.1\tools -I=D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\ --go_package=Microsoft.BI.Mediation.Schemas.Go --go_out=D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas.Go\Events\ D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\Events\App.proto
Unknown flag: --go_package

C:\Users\jingqicao>protoc.exe -I=C:\Users\jingqicao\.nuget\packages\google.protobuf.tools\3.6.1\tools -I=D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\ --go_package=Microsoft.BI.Mediation.Schemas.Go --go_out=D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas.Go\Events\ D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\Events\App.proto
Unknown flag: --go_package

C:\Users\jingqicao>protoc.exe -I=C:\Users\jingqicao\.nuget\packages\google.protobuf.tools\3.6.1\tools -I=D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\ --go_out=D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas.Go\Events\ D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\Events\App.proto
Events/App.proto:4:23: Invalid escape sequence in string literal.
Events/App.proto:4:31: Invalid escape sequence in string literal.
Events/App.proto:4:48: Invalid escape sequence in string literal.
Events/App.proto:4:56: Invalid escape sequence in string literal.
Events/App.proto:4:64: Invalid escape sequence in string literal.
Events/App.proto:6:1: Expected ";".

C:\Users\jingqicao>protoc.exe -I=C:\Users\jingqicao\.nuget\packages\google.protobuf.tools\3.6.1\tools -I=D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\ --go_out=D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas.Go\Events\ D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\Events\App.proto
Events/App.proto:4:23: Invalid escape sequence in string literal.
Events/App.proto:4:31: Invalid escape sequence in string literal.
Events/App.proto:4:48: Invalid escape sequence in string literal.
Events/App.proto:4:56: Invalid escape sequence in string literal.
Events/App.proto:4:64: Invalid escape sequence in string literal.

C:\Users\jingqicao>protoc.exe -I=C:\Users\jingqicao\.nuget\packages\google.protobuf.tools\3.6.1\tools -I=D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\ --go_out=D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas.Go\Events\ D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\Events\App.proto
Events/App.proto:4:23: Invalid escape sequence in string literal.
Events/App.proto:4:31: Invalid escape sequence in string literal.
Events/App.proto:4:48: Invalid escape sequence in string literal.
Events/App.proto:4:56: Invalid escape sequence in string literal.
Events/App.proto:4:64: Invalid escape sequence in string literal.
Events/App.proto:4:98: Invalid escape sequence in string literal.

C:\Users\jingqicao>protoc.exe -I=C:\Users\jingqicao\.nuget\packages\google.protobuf.tools\3.6.1\tools -I=D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\ --go_out=D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas.Go\Events\ D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\Events\App.proto
Events/App.proto:4:23: Invalid escape sequence in string literal.
Events/App.proto:4:26: Invalid escape sequence in string literal.
Events/App.proto:4:60: Invalid escape sequence in string literal.

C:\Users\jingqicao>protoc.exe -I=C:\Users\jingqicao\.nuget\packages\google.protobuf.tools\3.6.1\tools -I=D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\ --go_out=D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas.Go\Events\ D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\Events\App.proto

C:\Users\jingqicao>protoc.exe -I=C:\Users\jingqicao\.nuget\packages\google.protobuf.tools\3.6.1\tools -I=D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\ --go_out=D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas.Go\Events\ D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\Events\App.proto D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\Events\Auction.proto D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\Events\AuctionFacts.proto D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\Events\Bid.proto D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\Events\ClickEvent.proto D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\Events\CommonTypes.proto D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\Events\Device.proto D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\Events\ExternalUser.proto D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\Events\L1ClassifierScore.proto D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\Events\Location.proto D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\Events\MediationConfigParams.proto D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\Events\MediationServingLogRecord.proto D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\Events\OperatingSystem.proto D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\Events\PiiRecord.proto D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\Events\Revenue.proto D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\Events\Site.proto D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\Events\User.proto D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\Events\ViewabilityEvent.proto

Untitled

Posted on 2026-05-05

Untitled

Posted on 2026-05-05

spark.sessionState.conf.setConfString("spark.sql.streaming.minBatchesToRetain", "5")
logger.info("spark.sql.streaming.minBatchesToRetain:" + spark.sessionState.conf.minBatchesToRetain)
logger.info("spark.sql.shuffle.partitions:" + spark.sparkContext.getConf.get("spark.sql.shuffle.partitions"))
logger.info("spark.sql.streaming.multipleWatermarkPolicy:" + spark.sparkContext.getConf.get("spark.sql.streaming.multipleWatermarkPolicy"))
val logSparkConf = config.getConfig("logSparkConf")

tair使用规范

Posted on 2026-05-05 In tair

缓存Tair高性能使用规范

Untitled

Posted on 2026-05-05

Untitled

Posted on 2026-05-05

[toc]

文件间跳转

数学表达式

要启用这个功能，首先到Preference->Editor中启用。然后使用$符号包裹Tex命令，例如：$lim_{x \to \infty} \ exp(-x)=0$将产生如下的数学表达式：

下标

下标使用~~包裹，例如：H~~2~O将产生水的分子式。

上标

上标使用^包裹，例如：y^2^=4将产生表达式

插入表情

使用:happy:输入高兴的表情，使用:sad:输入悲伤的表情，使用:cry:输入哭的表情等。以此类推!

下划线

用HTML的语法Underline将产生下划线Underline.

删除线

GFM添加了删除文本的语法，这是标准的Markdown语法木有的。使用~~包裹的文本将会具有删除的样式，例如~~删除文本~将产生删除文本的样式。

代码

使用`包裹的内容将会以代码样式显示，例如

使用printf()

则会产生printf()样式。

输入~或者```然后回车，可以输入代码块，并且可以选择代码的语言。例如：public Class HelloWorld{ System.out.println(“Hello World!”); }

强调

使用两个*号或者两个_包裹的内容将会被强调。例如

*使用两个号强调内容** 使用两个下划线强调内容

将会输出

使用两个号强调内容使用两个下划线强调内容 Typroa 推荐使用两个号。

斜体

在标准的Markdown语法中，*和_包裹的内容会是斜体显示，但是GFM下划线一般用来分隔人名和代码变量名，因此我们推荐是用星号来包裹斜体内容。如果要显示星号，则使用转义：

*

插入图片

我们可以通过拖拉的方式，将本地文件夹中的图片或者网络上的图片插入。

插入URL连接

使用尖括号包裹的url将产生一个连接，例如：将产生连接:www.baidu.com.

如果是标准的url，则会自动产生连接，例如:www.baidu.com

目录列表Table of Contents(TOC)

输入[toc]然后回车，将会产生一个目录，这个目录抽取了文章的所有标题，自动更新内容。

水平分割线

使用***或者—，然后回车，来产生水平分割线。

标注

我们可以对某一个词语进行标注。例如

某些人用过了才知道[^注释] [^注释]:Somebody that I used to know.

将产生：

把鼠标放在注释上，将会有提示内容。

hive-官网索引

Posted on 2026-05-05 In hive

grouping sets 要保证sets中每个组合中缺失的维度，不能包含null值

group语法增强，可以使用维度下标表示，但如果grouping sets不支持，只能写字段名

hive-join

Posted on 2026-05-05 In hive

Join 类别

1. inner join

2. left/right join

3. cross join

4. left/right semi join 半开连接

LEFT SEMI JOIN:左半开连接会返回左边表的记录，前提是其记录对于右边表满足ON语句中的判定条件。对于常见的内连接（INNER JOIN），这是一个特殊的，优化了的情况。大多数的SQL方言会通过in…….exists结构来处理这种情况。

只会保留左/右表中能关联上的数据，且数量不会受另一张表影响而膨胀（即在原表中是几条数据，结果还是几条）

总结：

对待右表中重复key的处理方式差异：因为 left semi join 是 in(keySet) 的关系，遇到右表重复记录，左表会跳过，而 join on 则会一直遍历。
left semi join 中最后 select 的结果只许出现左表，因为右表只有 join key 参与关联计算了，而 join on 默认是整个关系模型都参与计算了。

5.

hive-官网索引

Posted on 2026-05-05 In hive

https://cwiki.apache.org/confluence/display/Hive/Home#Home-GeneralInformationaboutHive

https://cwiki.apache.org/confluence/display/Hive/LanguageManual

首页

语法手册

1. 命令行与客户端

1.1 数据类型

数据类型转换(Change Types)
数据类型隐式转换矩阵(Allowed Implicit Conversions)

1.2 参数配置

常用参数配置

1.3 select语法

All and DISTINCT 语句，UNION语句可以有相同效果
Partition Based Queries
HAVING Clause
REGEX Column Specification
GROUP BY
SORT/ORDER/CLUSTER/DISTRIBUTE BY
JOIN
UNION
TABLESAMPLE
Subqueries
Virtual Columns
Operators and UDFs hive函数
LATERAL VIEW
Windowing, OVER, and Analytics
Common Table Expressions 临时表语法

hive常用函数

Posted on 2026-05-05 In hive

常用

8位dt转10位

from_unixtime(unix_timestamp(t.dt,'yyyymmdd'),'yyyy-mm-dd')

8位减1小时

dt格式2019120100
from_unixtime(unix_timestamp(dt,'yyyyMMddHH') - 60*60, 'yyyy-MM-dd')

8位日期计算

regexp_replace(date_sub(from_unixtime(unix_timestamp(t.dt,'yyyymmdd'),'yyyy-mm-dd'), 10), '-', '')

当前日期前一天

保留2位小数

round(123.4567, 2)

时间戳转换

1
2

select hour(from_unixtime(event_timestamp/1000,'yyyy-MM-dd HH:mm:ss')) hour_time, count(distinct session_id)
from mart_eif_flow.bas_log_sdk_waimai_c where dt='20201217' group by hour(from_unixtime(event_timestamp/1000,'yyyy-MM-dd HH:mm:ss')) order by hour_time

计算

保留小数

字符串

替换

1	regexp_replace('abc/d', '/', '')

json

1 2	get_json_object(json_str, '$.csu_id') // json_str可以标准的json字符串，也可以是包含转译符的字符串