生而为人

hexo常见报错

Posted on 2026-08-03 In tools

bad indentation of a mapping entry

title中缩进不正常，均要用tab缩进

warning: adding embedded git repository: themes/next

warning: adding embedded git repository: themes/next
hint: You've added another git repository inside your current repository.
hint: Clones of the outer repository will not contain the contents of
hint: the embedded repository and will not know how to obtain it.
hint: If you meant to add a submodule, use:
hint:
hint: 	git submodule add <url> themes/next
hint:
hint: If you added this path by mistake, you can remove it from the
hint: index with:
hint:
hint: 	git rm --cached themes/next
hint:
hint: See "git help submodule" for more information.

git仓库包含子仓库时，add报错的解决办法

参考资料

hexo部署成功仍未更新或无法打开的情况

修改scheme后发布不起作用

Hexo博客 Next更换修改主题样式不生效的解决

Untitled

Posted on 2026-08-03

Microsoft Windows [Version 10.0.19044.2006]
(c) Microsoft Corporation. All rights reserved.

C:\Users\jingqicao>echo PATH
PATH

C:\Users\jingqicao>echo $PATH
$PATH

C:\Users\jingqicao>echo %PATH
%PATH

C:\Users\jingqicao>echo %PATH%
C:\Program Files (x86)\Common Files\Oracle\Java\javapath;C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\wbin;C:\Program Files\Common Files\Microsoft Shared\Microsoft Online Services;C:\Program Files (x86)\Common Files\Microsoft Shared\Microsoft Online Services;C:\windows\system32;C:\windows;C:\windows\System32\Wbem;C:\windows\System32\WindowsPowerShell\v1.0\;C:\windows\System32\OpenSSH\;C:\Program Files\Microsoft SQL Server\130\Tools\Binn\;C:\Program Files\Microsoft SQL Server\Client SDK\ODBC\170\Tools\Binn\;C:\Program Files\Git\cmd;C:\Program Files\Go\bin;D:\Support\gawk-3.1.6-1-bin\bin;D:\Support\Nuget;C:\Program Files\dotnet\;C:\Users\jingqicao\AppData\Local\CloudBuild;C:\Users\jingqicao\AppData\Local\CloudBuild;C:\Users\jingqicao\AppData\Local\Microsoft\WindowsApps;C:\Users\jingqicao\.dotnet\tools;C:\Users\jingqicao\AppData\Local\Programs\Microsoft VS Code\bin;C:\Users\jingqicao\Program Files\protoc-3.13.0-win64\bin;D:\Support\spark-3.0.1-bin-hadoop2.7\bin;D:\Support\spark-3.0.1-bin-hadoop2.7\sbin;C:\Program Files\Java\jdk1.8.0_291\bin;D:\Support\hadoop-2.7.7\bin;D:\Support\hadoop-2.7.7\sbin;D:\Support\apache-maven-3.8.1\bin;C:\Users\jingqicao\AppData\Local\Programs\Python\Python39\Scripts;C:\Users\jingqicao\AppData\Local\atom\bin;C:\Users\jingqicao\.dotnet\tools;D:\Support\Go\bin;

C:\Users\jingqicao>protoc.exe -I=D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\ --go_out=D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas.Go\Events\ --go_opt=file_extension=.pd.go D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\Events\App.proto
protoc-gen-go: no such flag -file_extension
--go_out: protoc-gen-go: Plugin failed with status code 1.

C:\Users\jingqicao>protoc.exe -I=D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\ --go_out=D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas.Go\Events\ --go_opt=.pd.go D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\Events\App.proto
protoc-gen-go: no such flag -.pd.go
--go_out: protoc-gen-go: Plugin failed with status code 1.

C:\Users\jingqicao>protoc.exe -I=D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\ --go_out=D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas.Go\Events\ --go_opt=. D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\Events\App.proto
protoc-gen-go: no such flag -.
--go_out: protoc-gen-go: Plugin failed with status code 1.

C:\Users\jingqicao>protoc.exe -I=D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\ --go_out=D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas.Go\Events\ D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\Events\App.proto
protoc-gen-go: unable to determine Go import path for "Events/App.proto"

Please specify either:
        • a "go_package" option in the .proto source file, or
        • a "M" argument on the command line.

See https://developers.google.com/protocol-buffers/docs/reference/go-generated#package for more information.

--go_out: protoc-gen-go: Plugin failed with status code 1.

C:\Users\jingqicao>protoc.exe -I=D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\ --go_out=D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas.Go\Events\ D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\Events\App.proto D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\Events\Auction.proto D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\Events\AuctionFacts.proto D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\Events\Bid.proto D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\Events\ClickEvent.proto D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\Events\CommonTypes.proto D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\Events\Device.proto D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\Events\ExternalUser.proto D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\Events\L1ClassifierScore.proto D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\Events\Location.proto D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\Events\MediationConfigParams.proto D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\Events\MediationServingLogRecord.proto D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\Events\OperatingSystem.proto D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\Events\PiiRecord.proto D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\Events\Revenue.proto D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\Events\Site.proto D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\Events\User.proto D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\Events\ViewabilityEvent.proto
google/protobuf/timestamp.proto: File not found.
Events/ViewabilityEvent.proto:4:1: Import "google/protobuf/timestamp.proto" was not found or had errors.
Events/ViewabilityEvent.proto:21:5: "google.protobuf.Timestamp" is not defined.
Events/ClickEvent.proto:4:1: Import "google/protobuf/timestamp.proto" was not found or had errors.
Events/ClickEvent.proto:18:9: "google.protobuf.Timestamp" is not defined.
Events/AuctionFacts.proto:3:1: Import "google/protobuf/timestamp.proto" was not found or had errors.
Events/AuctionFacts.proto:10:1: Import "Events/ViewabilityEvent.proto" was not found or had errors.
Events/AuctionFacts.proto:11:1: Import "Events/ClickEvent.proto" was not found or had errors.
Events/AuctionFacts.proto:23:9: "google.protobuf.Timestamp" is not defined.
Events/AuctionFacts.proto:45:9: "ViewabilityEvent" is not defined.
Events/AuctionFacts.proto:46:9: "ViewabilityEvent" is not defined.
Events/AuctionFacts.proto:47:9: "ClickEvent" is not defined.

C:\Users\jingqicao>protoc.exe -I=D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\ -I=D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\ --go_out=D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas.Go\Events\ D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\Events\App.proto
protoc-gen-go: unable to determine Go import path for "Events/App.proto"

Please specify either:
        • a "go_package" option in the .proto source file, or
        • a "M" argument on the command line.

See https://developers.google.com/protocol-buffers/docs/reference/go-generated#package for more information.

--go_out: protoc-gen-go: Plugin failed with status code 1.

C:\Users\jingqicao>protoc.exe -I=C:\Users\jingqicao\.nuget\packages\google.protobuf.tools\3.6.1\tools -I=D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\ --go_out=D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas.Go\Events\ D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\Events\App.proto
protoc-gen-go: unable to determine Go import path for "Events/App.proto"

Please specify either:
        • a "go_package" option in the .proto source file, or
        • a "M" argument on the command line.

See https://developers.google.com/protocol-buffers/docs/reference/go-generated#package for more information.

--go_out: protoc-gen-go: Plugin failed with status code 1.

C:\Users\jingqicao>protoc.exe -I=C:\Users\jingqicao\.nuget\packages\google.protobuf.tools\3.6.1\tools -I=D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\ --go_package=Microsoft.BI.Mediation.Schemas.Go --go_out=D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas.Go\Events\ D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\Events\App.proto
Unknown flag: --go_package

C:\Users\jingqicao>protoc.exe -I=C:\Users\jingqicao\.nuget\packages\google.protobuf.tools\3.6.1\tools -I=D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\ --go_package=Microsoft.BI.Mediation.Schemas.Go --go_out=D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas.Go\Events\ D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\Events\App.proto
Unknown flag: --go_package

C:\Users\jingqicao>protoc.exe -I=C:\Users\jingqicao\.nuget\packages\google.protobuf.tools\3.6.1\tools -I=D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\ --go_out=D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas.Go\Events\ D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\Events\App.proto
Events/App.proto:4:23: Invalid escape sequence in string literal.
Events/App.proto:4:31: Invalid escape sequence in string literal.
Events/App.proto:4:48: Invalid escape sequence in string literal.
Events/App.proto:4:56: Invalid escape sequence in string literal.
Events/App.proto:4:64: Invalid escape sequence in string literal.
Events/App.proto:6:1: Expected ";".

C:\Users\jingqicao>protoc.exe -I=C:\Users\jingqicao\.nuget\packages\google.protobuf.tools\3.6.1\tools -I=D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\ --go_out=D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas.Go\Events\ D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\Events\App.proto
Events/App.proto:4:23: Invalid escape sequence in string literal.
Events/App.proto:4:31: Invalid escape sequence in string literal.
Events/App.proto:4:48: Invalid escape sequence in string literal.
Events/App.proto:4:56: Invalid escape sequence in string literal.
Events/App.proto:4:64: Invalid escape sequence in string literal.

C:\Users\jingqicao>protoc.exe -I=C:\Users\jingqicao\.nuget\packages\google.protobuf.tools\3.6.1\tools -I=D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\ --go_out=D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas.Go\Events\ D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\Events\App.proto
Events/App.proto:4:23: Invalid escape sequence in string literal.
Events/App.proto:4:31: Invalid escape sequence in string literal.
Events/App.proto:4:48: Invalid escape sequence in string literal.
Events/App.proto:4:56: Invalid escape sequence in string literal.
Events/App.proto:4:64: Invalid escape sequence in string literal.
Events/App.proto:4:98: Invalid escape sequence in string literal.

C:\Users\jingqicao>protoc.exe -I=C:\Users\jingqicao\.nuget\packages\google.protobuf.tools\3.6.1\tools -I=D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\ --go_out=D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas.Go\Events\ D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\Events\App.proto
Events/App.proto:4:23: Invalid escape sequence in string literal.
Events/App.proto:4:26: Invalid escape sequence in string literal.
Events/App.proto:4:60: Invalid escape sequence in string literal.

C:\Users\jingqicao>protoc.exe -I=C:\Users\jingqicao\.nuget\packages\google.protobuf.tools\3.6.1\tools -I=D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\ --go_out=D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas.Go\Events\ D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\Events\App.proto

C:\Users\jingqicao>protoc.exe -I=C:\Users\jingqicao\.nuget\packages\google.protobuf.tools\3.6.1\tools -I=D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\ --go_out=D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas.Go\Events\ D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\Events\App.proto D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\Events\Auction.proto D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\Events\AuctionFacts.proto D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\Events\Bid.proto D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\Events\ClickEvent.proto D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\Events\CommonTypes.proto D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\Events\Device.proto D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\Events\ExternalUser.proto D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\Events\L1ClassifierScore.proto D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\Events\Location.proto D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\Events\MediationConfigParams.proto D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\Events\MediationServingLogRecord.proto D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\Events\OperatingSystem.proto D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\Events\PiiRecord.proto D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\Events\Revenue.proto D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\Events\Site.proto D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\Events\User.proto D:\project\Ads.BI.Mediation\private\Schemas\Microsoft.BI.Mediation.Schemas\PROTO\Events\ViewabilityEvent.proto

Untitled

Posted on 2026-08-03

spark.sessionState.conf.setConfString("spark.sql.streaming.minBatchesToRetain", "5")
logger.info("spark.sql.streaming.minBatchesToRetain:" + spark.sessionState.conf.minBatchesToRetain)
logger.info("spark.sql.shuffle.partitions:" + spark.sparkContext.getConf.get("spark.sql.shuffle.partitions"))
logger.info("spark.sql.streaming.multipleWatermarkPolicy:" + spark.sparkContext.getConf.get("spark.sql.streaming.multipleWatermarkPolicy"))
val logSparkConf = config.getConfig("logSparkConf")

Untitled

Posted on 2026-08-03

hive-官网索引

Posted on 2026-08-03 In hive

grouping sets 要保证sets中每个组合中缺失的维度，不能包含null值

group语法增强，可以使用维度下标表示，但如果grouping sets不支持，只能写字段名

hive-join

Posted on 2026-08-03 In hive

Join 类别

1. inner join

2. left/right join

3. cross join

4. left/right semi join 半开连接

LEFT SEMI JOIN:左半开连接会返回左边表的记录，前提是其记录对于右边表满足ON语句中的判定条件。对于常见的内连接（INNER JOIN），这是一个特殊的，优化了的情况。大多数的SQL方言会通过in…exists结构来处理这种情况。

只会保留左/右表中能关联上的数据，且数量不会受另一张表影响而膨胀（即在原表中是几条数据，结果还是几条）

总结：

对待右表中重复key的处理方式差异：因为 left semi join 是 in(keySet) 的关系，遇到右表重复记录，左表会跳过，而 join on 则会一直遍历。
left semi join 中最后 select 的结果只许出现左表，因为右表只有 join key 参与关联计算了，而 join on 默认是整个关系模型都参与计算了。

5.

hive-官网索引

Posted on 2026-08-03 In hive

https://cwiki.apache.org/confluence/display/Hive/Home#Home-GeneralInformationaboutHive

https://cwiki.apache.org/confluence/display/Hive/LanguageManual

首页

语法手册

1. 命令行与客户端

1.1 数据类型

数据类型转换(Change Types)
数据类型隐式转换矩阵(Allowed Implicit Conversions)

1.2 参数配置

常用参数配置

1.3 select语法

All and DISTINCT 语句，UNION语句可以有相同效果
Partition Based Queries
HAVING Clause
REGEX Column Specification
GROUP BY
SORT/ORDER/CLUSTER/DISTRIBUTE BY
JOIN
UNION
TABLESAMPLE
Subqueries
Virtual Columns
Operators and UDFs hive函数
LATERAL VIEW
Windowing, OVER, and Analytics
Common Table Expressions 临时表语法

hive常用函数

Posted on 2026-08-03 In hive

常用

8位dt转10位

from_unixtime(unix_timestamp(t.dt,'yyyymmdd'),'yyyy-mm-dd')

8位减1小时

dt格式2019120100
from_unixtime(unix_timestamp(dt,'yyyyMMddHH') - 60*60, 'yyyy-MM-dd')

8位日期计算

regexp_replace(date_sub(from_unixtime(unix_timestamp(t.dt,'yyyymmdd'),'yyyy-mm-dd'), 10), '-', '')

当前日期前一天

保留2位小数

round(123.4567, 2)

时间戳转换

1
2

select hour(from_unixtime(event_timestamp/1000,'yyyy-MM-dd HH:mm:ss')) hour_time, count(distinct session_id)
from mart_eif_flow.bas_log_sdk_waimai_c where dt='20201217' group by hour(from_unixtime(event_timestamp/1000,'yyyy-MM-dd HH:mm:ss')) order by hour_time

计算

保留小数

字符串

替换

1	regexp_replace('abc/d', '/', '')

json

1 2	get_json_object(json_str, '$.csu_id') // json_str可以标准的json字符串，也可以是包含转译符的字符串

hive常用函数

Posted on 2026-08-03 In hive

with temp_table as (
  select '1' as f1, null as f2
  union all
  select '1', '3'
)

count distinct 结果是字段均不为null的去重值

所以在通过count distinct判断字段的重复情况时，注意把相关字段的null附上值

select count(*)
from temp_table
--
`count`(*)
2
    
select count(distinct f1, f2)
from temp_table
--
`count`(DISTINCT `f1`, `f2`)
1

select count(f1, f2)
from temp_table
--
`count`(`f1`, `f2`)
1

关联字段中有null时是关联不上的，即使左右都是null

select a.f1, a.f2, b.f1, b.f2
from temp_table a
left join temp_table b
on a.f1 = b.f1
and a.f2 = b.f2
--
f1	f2	f1	f2
1	NULL	NULL	NULL
1	3	1	3

sum中包含null，不会影响sum的值

select sum(ct)
from(select 1 as ct
union all
select null)a

--
`sum`(`ct`)
1

Untitled

Posted on 2026-08-03

Hive下查看表占用空间大小的方法